Control apparatus, movable apparatus, and remote-control system

ABSTRACT

A control apparatus for controlling a movable apparatus including a first imaging apparatus for imaging a target to acquire a first image and a second imaging apparatus for imaging a part of the target to acquire a second image includes circuitry configured to acquire, from the movable apparatus, state information indicating a movement state of the movable apparatus, receive the first image and the second image from the movable apparatus based on the acquired state information, and output the first image and the second image selectively based on the acquired state information.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority pursuant to 35 U.S.C. § 119(a) toJapanese Patent Application No. 2018-123655, filed on Jun. 28, 2018 inthe Japan Patent Office, the disclosure of which is incorporated byreference herein in its entirety.

BACKGROUND Technical Field

This disclosure relates to a control apparatus, a display terminal, aninformation processing apparatus, a movable apparatus, a remote-controlsystem, an output control method, and an imaging control apparatus.

Background Art

Remote-control systems that remotely operate one or more robots(hereinafter, referred to as robot or robots) disposed at one or moreremote sites or locations are known as the systems to remotely operatethe robots using a display terminal located at another remote site(e.g., control center) communicable with the robots via a communicationnetwork. The remote-control system can check or confirm information ofeach site where each robot is located by displaying images, captured byan imaging apparatus provided for each robot, on the display terminaldisposed at the remote site (e.g., control center).

Further, technologies using robots equipped with different types ofimaging apparatuses or devices to accurately confirm situations aroundthe robots are also known. For example, one technology discloses awirelessly-controllable movable apparatus equipped with a front camerafor capturing images of the front region of the movable apparatus, arear camera for capturing images of the rear region of the movableapparatus, and a full-view camera that can capture the entire perimeteraround the movable apparatus.

SUMMARY

As one aspect of the present invention, a control apparatus forcontrolling a movable apparatus including a first imaging apparatus forimaging a target to acquire a first image and a second imaging apparatusfor imaging a part of the target to acquire a second image is devised.The control apparatus for controlling the movable apparatus includescircuitry configured to acquire, from the movable apparatus, stateinformation indicating a movement state of the movable apparatus,receive the first image and the second image from the movable apparatusbased on the acquired state information, and output the first image andthe second image selectively based on the acquired state information.

As another aspect of the present invention, a movable apparatus isdevised. The movable apparatus includes a movement mechanism configuredto move the movable apparatus, a first imaging apparatus for imaging atarget to acquire a first image, a second imaging apparatus for imaginga part of the target to acquire a second image, and circuitry configuredto acquire state information indicating a movement state of the movableapparatus, acquire the first image using the first imaging apparatus andthe second image using the second imaging apparatus, and output theacquired first image and second image to a display terminal,communicable with the circuitry, based on the acquired stateinformation.

As another aspect of the present invention, a remote-control system isdevised. The remote-control system includes a movable apparatusincluding a first imaging apparatus for imaging a target to acquire afirst image and a second imaging apparatus for imaging a part of thetarget to acquire a second image, a display terminal communicable withthe movable apparatus via a communication network for remotely operatingthe movable apparatus, and circuitry configured to acquire, from themovable apparatus, state information indicating a movement state of themovable apparatus, receive the first image and the second image from themovable apparatus based on the acquired state information, and output,to the display terminal, the first image and the second imageselectively based on the acquired state information.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the description and many of theattendant advantages and features thereof can be readily acquired andunderstood from the following detailed description with reference to theaccompanying drawings, wherein:

FIG. 1 illustrates an example of a system configuration of aremote-control system according to an embodiment;

FIG. 2 illustrates an example of a schematic configuration of a robotaccording to an embodiment;

FIGS. 3A to 3C (FIG. 3) illustrate another example of schematicconfigurations of a robot of variant example 1-1 according to anembodiment;

FIGS. 4A and 4B (FIG. 4) illustrate another example of schematicconfigurations of a robot of variant example 1-2 according to anembodiment;

FIGS. 5A and 5B (FIG. 5) illustrate another example of schematicconfigurations of a robot of variant example 2-1 according to anembodiment;

FIG. 6 illustrates another example of a schematic configuration of arobot of variant example 2-2 according to an embodiment;

FIG. 7 illustrates another example of a schematic configuration of arobot according of variant example 3 according to an embodiment;

FIG. 8A illustrates an example of a hemispherical image (at front side)captured by a special imaging apparatus;

FIG. 8B illustrates an example of a hemispherical image (at rear side)captured by the special imaging apparatus;

FIG. 8C illustrates an example of an image expressed by theequirectangular projection method;

FIG. 9A is a conceptual diagram illustrating a state in which a sphereis covered with an equirectangular projection image;

FIG. 9B illustrates an example of a full-view spherical image;

FIG. 10 illustrates a position of a virtual camera and a position of aspecific region when a full-view spherical image corresponds to athree-dimensional stereoscopic sphere;

FIG. 11A is a perspective view of a virtual camera and athree-dimensional stereoscopic sphere of FIG. 10;

FIG. 11B illustrates an example of a specific region image whendisplayed on a display of a display terminal;

FIG. 12 illustrates a relationship between specific region informationand an image of a specific region;

FIG. 13 illustrates an example of a hardware block diagram of a robotaccording to an embodiment;

FIG. 14 illustrates an example of a hardware block diagram of a displayterminal according to an embodiment;

FIG. 15 illustrates an example of a hardware block diagram of amanagement server (control server) according to an embodiment;

FIGS. 16A and 16B (FIG. 16) illustrate an example of a functional blockdiagram of a remote-control system according to a first embodiment;

FIG. 17A illustrates an example of a command table according to a firstembodiment;

FIG. 17B illustrates an example of an imaging parameter table accordingto a first embodiment;

FIG. 18 illustrates an example of a state management table (statecontrol table) according to a first embodiment;

FIG. 19 illustrates an example of a condition table according to a firstembodiment;

FIG. 20 illustrates an example of a user command table according to afirst embodiment;

FIG. 21A is an example of an authentication management DB(authentication control DB);

FIG. 21B is an example of a terminal management DB (terminal controlDB);

FIG. 22A is an example of a destination list management DB (destinationlist control DB);

FIG. 22B is an example of a session management DB (session control DB);

FIG. 23 is an example of a sequence diagram illustrating a preparatorystage for starting data transmission and reception between a robot and adisplay terminal;

FIG. 24 illustrates an example of a destination list screen displayed ona display terminal according to a first embodiment;

FIG. 25 is an example of a sequence diagram illustrating processing fromselecting a destination candidate to starting transmission and receptionof image data;

FIG. 26 is an example of a sequence diagram illustrating a transmissionprocess of various data from a robot to a display terminal in aremote-control system according to a first embodiment;

FIG. 27 illustrates an example of a display screen displayed on adisplay terminal according to a first embodiment;

FIG. 28 illustrates an example of state information according to a firstembodiment;

FIGS. 29A and 29B (FIG. 29) illustrate examples of a display screendisplayed on a display terminal when a robot is moving in a forwarddirection;

FIG. 30 illustrates another example of a display screen displayed on adisplay terminal when a robot is moving in a forward direction;

FIG. 31 is an example of a flowchart illustrating a robot controlprocess based on a movement state of a robot using a display terminalaccording to a first embodiment;

FIG. 32 is an example of a flowchart illustrating a robot controlprocess based on an input command at a display terminal according to afirst embodiment;

FIG. 33 is an example of a flowchart illustrating a robot controlprocess based on a request command transmitted from a display terminalaccording to according to a first embodiment;

FIG. 34 is an example of a sequence diagram illustrating a process ofdisplaying a detailed image in a remote-control system according to afirst embodiment;

FIG. 35 illustrates an example of a display screen displaying detailedimage data transmitted from a robot;

FIG. 36 illustrates an example of a display screen displaying aline-of-sight position (viewing position) of an operator on a displayterminal.

FIG. 37 is an example of a sequence diagram illustrating a process ofswitching an image displayed on a display terminal in an environment ofa remote-control system according to a first embodiment;

FIG. 38 is an example of a display screen displayed on a head-mountdisplay used as an example of a display terminal;

FIG. 39 is an example of a sequence diagram illustrating a process ofswitching an image displayed on a display terminal in an environment ofa remote-control system according to a second embodiment.

FIG. 40 illustrates an example of a system configuration of aremote-control system according to a third embodiment;

FIGS. 41A and 41B (FIG. 41) illustrate an example of a functional blockdiagram of a remote-control system according to a third embodiment;

FIG. 42 is an example of a sequence diagram illustrating processing whena robot moves in an environment of a remote-control system according toa third embodiment.

FIG. 43 is an example: of a flowchart illustrating image processing on afull-view spherical image data based on the movement state of a robot inan image processing server according to a third embodiment; and

FIG. 44 is an example of a sequence diagram illustrating a process ofswitching an image displayed on a display terminal in an environment ofa remote-control system according to a third embodiment.

The accompanying drawings are intended to depict embodiments of thepresent invention and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted.

DETAILED DESCRIPTION

A description is now given of exemplary embodiments of the presentinventions. It should be noted that although such terms as first,second, etc. may be used herein to describe various elements,components, regions, layers and/or units, it should be understood thatsuch elements, components, regions, layers and/or units are not limitedthereby because such terms are relative, that is, used only todistinguish one element, component, region, layer or unit from anotherregion, layer or unit. Thus, for example, a first element, component,region, layer or unit discussed below could be termed a second element,component, region, layer or unit without departing from the teachings ofthe present inventions.

In addition, it should be noted that the terminology used herein is forthe purpose of describing particular embodiments only and is notintended to be limiting of the present inventions. Thus, for example, asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. Moreover, the terms “include” and/or “including,” when usedin this specification, specify the presence of stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof.

Hereinafter, a description is given of a configuration for carrying outone or more embodiments of the present invention with reference to thedrawings. In the description of the drawings, the same elements aredenoted by the same reference numerals, and duplicated descriptions maybe omitted.

In conventional technologies, images acquired by different cameras(i.e., imaging apparatuses) provided for a robot (i.e., movableapparatus) are output without an effective method, with which thecontrol of the movable apparatus may not be performed with higherprecision depending on methods.

System Configuration:

FIG. 1 illustrates an example of a system configuration of aremote-control system 1 a according to an embodiment. The remote-controlsystem 1 a illustrated in FIG. 1 can remotely control one or more robots10 located at one or more sites, such as remote sites, using a displayterminal 50 to perform the operation management, maintenance operationsor the like for devices or apparatuses disposed at one or more sites andto confirm positions and movement lines of persons existing in one ormore sites.

As illustrated in FIG. 1, the remote-control system 1 a includes, forexample, a plurality of robots 10 (robots 10A, 10B, 10C) disposed ineach of a plurality of sites (sites A, B, C), the display terminal 50,and a management server (control server) 90. Hereinafter, the robots10A, 1013, and 10C may be simply referred to the robot 10 if thedistinction of each robot is not required. In this description, the termof “manage” and “management” are also referred to as “control” and“controlling,” As illustrated in FIG. 1, the robot 10, the displayterminal 50 and the management server 90 are communicably connected toeach other via a communication network 9. The communication network 9employs, for example, a local area network (LAN), a dedicated line, theInternet, and the like. The communication network 9 can use a wirelessconnection communication, such as Wi-Fi (registered trademark), inaddition to the wired connection communication.

The robot 10, disposed at each site (site A, site B, site C), is anexample of a movable apparatus that can be autonomously driven under ana remote-control using, for example, the display terminal 50. The robot10 can move in each site while capturing images (first image) of one ormore targets (e.g., objects) in a wide range using a special imagingapparatus 21 (first imaging apparatus), which will be described later,and transmits the images acquired by the special imaging apparatus 21 tothe display terminal 50 to provide information (e.g., images) in eachsite to an operator who operates the robot 10 using the display terminal50.

Further, the robot 10 can acquire images of a part or portion of thetargets e.g., objects), captured by the special imaging apparatus 21, asdetailed images (second image), using a general imaging apparatus 23(second imaging apparatus), which will be described later, and transmitsthe detailed images acquired by the general imaging apparatus 23 to thedisplay terminal 50 to provide detailed information (e.g., images) in aspecific region in each site to the operator who operates the robot 10using the display terminal 50.

In this description, the robot 10 is an example of movable apparatus ormachine, the special imaging apparatus 21 is used as first imagingapparatus, the general imaging apparatus 23 is used as second imagingapparatus, the image captured by the special imaging apparatus 21 (firstimaging apparatus) is referred to as the first image, and the imagecaptured by the general imaging apparatus 23 (second imaging apparatus)is referred to as the second image.

The display terminal 50 is a terminal device or apparatus, such as apersonal computer (PC) that can be used to perform a remote control ofthe robot 10 disposed at each site (site A, site B, site C). The displayterminal 50 can display images, for example, full-view spherical imagesand/or detailed images transmitted from the robot 10. Then, the operatorcan perform the remote control of the robot 10 by viewing the imagesdisplayed on the display terminal 50.

The display terminal 50 may be provided with a display device used fordisplaying the images transmitted from the robot 10. The displayterminal 50 can be a tablet terminal, a cellular phone, a smart phone, awearable terminal such as a head-mounted display (HIVED), and acommunication terminal such as a personal digital assistant (PDA)equipped with a wide-angle screen (e.g., cylinder screen, full-viewspherical screen, a half spherical screen).

The management server 90 is a control server used for managing orcontrolling communication between the display terminal 50 and the robot10 located at each site. The management server 90 can be also referredto as the control server 90. The management server 90 can be connectedto the robot 10 and the display terminal 50 via the communicationnetwork 9. It should be noted that the management server 90 can beconfigured by a plurality of servers, in which any server can beprovided with any functions.

The site where the robot 10 is disposed includes, for example, indoorsites, such as offices, warehouses, factories, schools, and the like,and outdoor sites, such as construction sites, and the like. Theoperator who operates the robot 10 using the display terminal 50 canconfirm the position and the movement line of persons existing in thesite and perform the management operation and maintenance operation ofdevices or apparatuses disposed at the site by checking the capturedimages of the site transmitted from the robot 10. Further, the robot 10and the display terminal 50 can also perform bidirectional communication(remote communication conference) by transmitting and receiving thecaptured images between the robot 10 and the display terminal 50.

In the configuration in FIG. 1, one robot 10 is disposed in each site,but a plurality of robots 10 can be disposed at each one site. Further,the display terminal 50 can be configured to communicate with each ofthe robots 10 disposed in a plurality of sites or can be configured tocommunicate with only the robot 10 disposed in one site.

Configuration of Robot:

Hereinafter, a description is given of a configuration the robot 10 ofFIG. 1 with reference to FIGS. 2 to 9. FIG. 2 illustrates an example ofa structure of the robot 10 according to the embodiment. As illustratedin FIG. 2, the robot 10 includes, for example, a movement mechanism 17,a housing 15, a manipulation arm 11, a rotary shaft 12, an image capturemechanism 20, and a mounting member 25. The movement mechanism 17 isused for moving the robot 10. The housing 15 includes a control device30 (FIG. 13) used for controlling processing and operation of the robot10. The rotary shaft 12, such as a joint, is used for rotating(transforming) the manipulation arm 11. The mounting member 25 isconnected to the manipulation arm 11 and supports the image capturemechanism 20.

As illustrated in FIG. 2, the image capture mechanism 20 includes, forexample, the special imaging apparatus 21 (first imaging apparatus) andthe general imaging apparatus 23 (second imaging apparatus).

The special imaging apparatus 21 can capture images of targets (e.g.,objects), such as persons, physical objects, landscape, and the like andacquire special images (first image), such as a panoramic image orfill-view spherical image (i.e., 360-degree image). The general imagingapparatus 23 captures images of a part or portion of the targets (e.g.,objects), captured by the special imaging apparatus 21, to acquire thedetailed images (second image) of the targets (e.g., objects). Thespecial imaging apparatus 21 is, for example, a special digital camerafor capturing images of the targets (e.g., objects, such as twohemispherical images used as the source of the full-view spherical image(panorama image) used as the first image. Further, the first image isnot limited to the full-view spherical image, but can be any imagecapturing a relatively wide range such as the wide-angle image, whichcan be used to check or confirm a relatively wider area around the robot10.

The general imaging apparatus 23 is, for example, a digital camera, suchas a digital single lens reflex (SLR) camera, a compact digital camera,or the like capable of acquiring planar images (detailed image) used asthe second image. The second image is an image capturing a relativelysmaller range, which can be used to check or confirm a relativelynarrower area or focused area around the robot 10.

The details of the full-view spherical image captured by the specialimaging apparatus 21 and the hemispherical image used as the source ofthe full-view spherical image will be described later (see FIGS. 8 to12). In this description, the special imaging apparatus 21 is an exampleof the first imaging apparatus or device, and the general imagingapparatus 23 is an example of the second imaging apparatus or device.Hereinafter, the target to be captured may be also referred to as theobject.

The robot 10 is configured to transmit full-view spherical image data200, which is the full-view spherical image acquired by the specialimaging apparatus 21, to the display terminal 50. The image of thefull-view spherical image data 200 can be still images or video images,and can be both of video images and still images. The full-viewspherical image data 200 can further include audio data together withthe image data.

The image acquired by the special imaging apparatus 21 is not limited tothe full-view spherical image but can be a wide-angle image having anangle of view equal to or greater than a specific value of angle ofview. In this case, the wide-angle image is acquired by the specialimaging apparatus 21, such as a wide-angle camera or a stereo camera.Specifically, the special imaging apparatus 21 is an imaging unitcapable of acquiring the image (e.g., full-view spherical image andwide-angle image) captured by using a lens having a focal length shorterthan a specific value of focal length. The image (e.g., full-viewspherical image, wide-angle image) acquired by the special imagingapparatus 21 is an example of the first image in this description. Ineach of the following description, the image acquired by the specialimaging apparatus 21 is assumed to be, for example, the full-viewspherical image.

Further, the robot 10 is configured to transmit detailed image data 250,which is a detailed image acquired by the general imaging apparatus 23,to the display terminal 50. The detailed image acquired by the generalimaging apparatus 23 is an image acquired by imaging a part or portionof an object, which is captured by the special imaging apparatus 21,with an angle of view equal to or greater than a specific value. Thatis, the general imaging apparatus 23 is an imaging unit capable ofacquiring the image (detailed image) captured by using a lens having afocal length longer than the focal length of the lens of the specialimaging apparatus 21. The image acquired by the general imagingapparatus 23 (detailed image and planar image) is an example of thesecond image in this description.

Specifically, when the robot 10 is moved using the remote controlperformed by the operator of the robot 10, the display terminal 50displays the full-view spherical image, which can view a wide range ofthe circumference or surroundings of the robot 10. Further, if theoperator of the robot 10 wants to confirm detailed information about aspecific region included in the full-view spherical image, the displayterminal 50 displays the detailed image acquired by the general imagingapparatus 23. In this configuration, the special imaging apparatus 21 isone imaging unit used for performing imaging processing to acquire theimage (e.g., full-view spherical image and wide angle image) to be usedby the operator of the robot 10 for confirming or checking thecircumference or surroundings of the robot 10, and the general imagingapparatus 23 is another imaging unit used for performing imagingprocessing to acquire the image (e.g., detailed image) to be used by theoperator of the robot 10 for confirming or checking a state of aspecific region around the robot 10. With this configuration, thedisplay terminal 50 can change or switch a display of the full-viewspherical image (first image) and the detailed image (second image) toimprove the operability of the robot 10 by the operator.

Hereinafter, it is assumed that the image capture mechanism 20 includesthe special imaging apparatus 21 and the general imaging apparatus 23 asseparate imaging apparatuses, but the functions of the special imagingapparatus 21 and the general imaging apparatus 23 can be implemented bya single imaging apparatus.

The movement mechanism 17 is a unit for moving the robot 10, andincludes, for example, one or more wheels, a drive motor, a driveencoder, a steering motor, a steering encoder, and the like. Since themovement control of the robot 10 is known technology, the detaileddescription is omitted. Typically, the robot 10 receives a travelinstruction from the operator (e.g., display terminal 50), and then themovement mechanism 17 moves the robot 10 based on the received travelinstruction.

Hereinafter, the movement mechanism 17 is assumed to include two wheels,but the movement mechanism 17 can employ any mechanisms, such as atwo-leg walking type and a single wheel. Further, the shape of the robot10 is not limited to a vehicle type illustrated in FIG. 2, but can be,for example, a humanoid type of two legs, a form of reproducing ananimal form, a form of reproducing a specific character, or the

The housing 15 is disposed at a body portion of the robot 10, andincludes, for example, a power supply unit for supplying power necessaryfor the robot 10 and the control device 30 for controlling theprocessing or operation of the robot 10.

The manipulation arm 11 is a movable member used for adjusting the imagecapture position of the special imaging apparatus 21 and the generalimaging apparatus 23 disposed on the mounting member 25. Themanipulation arm 11 can be rotated using the rotary shaft 12 to changethe orientation of the special imaging apparatus 21 and the generalimaging apparatus 23. The robot 10 can change the image capturedirection of the image capture mechanism 20 (i.e., first image capturedirection used for the special imaging apparatus 21 and second imagecapture direction used for the general imaging apparatus 23) by changingthe direction of the robot 10 by the movement mechanism 17 and byrotating or transforming the manipulation arm 11.

In addition to the above described configuration, the robot 10 mayinclude various sensors capable of detecting the information around therobot 10. The sensors are, for example, sensor devices such asbarometers, thermometers, photometers, human sensory sensors, andilluminance meters. Further, in addition to the image capture mechanism20 disposed on the manipulation arm 11, the robot 10 can include anoperation unit enabling an additional operation of the robot 10 otherthan the movement of the robot 10. The operation unit is, for example, ahand that can grip an object.

Variance of Robot: Variant Example 1 of Robot:

Hereinafter, a description is given of a configuration of the robot 10of variant examples 1-1 and 1-2 with reference to FIG. 3 and FIG. 4. Therobots 10 illustrated in FIGS. 3A to 3C (FIG. 3) differ from theconfiguration illustrated in FIG. 2 in the arrangement of the specialimaging apparatus 21 and the image capture mechanism 20.

As described with reference to FIG. 2, since the special imagingapparatus 21 and the general imaging apparatus 23 are different inimaging purposes, it is preferable to change the arrangement of thespecial imaging apparatus 21 and the general imaging apparatus 23according to the imaging purposes.

In an image capture mechanism 20 a illustrated in FIG. 3A, the specialimaging apparatus 21 is disposed on the upper part of the generalimaging apparatus 23. The special imaging apparatus 21 is used tocapture a wide range of the surroundings around the robot 10. Therefore,if the robot 10 has the configuration of the image capture mechanism 20a arranged as illustrated in FIG. 3A, the special imaging apparatus 21and the general imaging apparatus 23 can be effectively used separately.

In an image capture mechanism 20 b illustrated in FIG. 3B, the specialimaging apparatus 21 is disposed at the rear of the general imagingapparatus 23. The region in the front direction (second image capturedirection) of the general imaging apparatus 23 is a region where theoperator of the robot 10 wants to check in detail. Therefore, byarranging the image capture mechanism 20 b as illustrated in FIG. 3B,the general imaging apparatus 23 can capture images of a region in thefront direction (second image capture direction) without interference ofobstacles or the like. Further, by arranging the image capture mechanism20 b as illustrated in FIG. 3B, the special imaging apparatus 21 cancapture images of a region where the general imaging apparatus 23 cannotcapture images (e.g., rear region of the general imaging apparatus 23)with a relatively good resolution without capturing an image of thegeneral imaging apparatus 23.

Further, in an image capture mechanism 20 c illustrated in FIG. 3C, thespecial imaging apparatus 21 is disposed at the lower part of thegeneral imaging apparatus 23. The state of the ground (foot area)becomes important when the robot 10 is moved. Therefore, by arrangingthe image capture mechanism 20 c as illustrated in FIG. 3C, the specialimaging apparatus 21 can capture images of the ground (foot area)without being obstructed by the general imaging apparatus 23 and/or themounting member 25. With this configuration, the operator of the robot10 can move the robot 10 more safely by viewing the full-view sphericalimage acquired by the special imaging apparatus 21.

FIGS. 4A and 4B (FIG. 4) illustrate example of structures of themanipulation arm 11 of the robot 10 different from the structureillustrated in FIG. 2. It is preferable that the manipulation arm 11 cansecure a given movable range, which can change depending on applicationsof the robot 10. In a case of FIG. 4A, a manipulation arm 11 a has nojoint members, and the direction or orientation of the manipulation arm11 a can be changed by the rotary shaft 12. If the height and distanceof a portion to be captured by the special imaging apparatus 21 or thegeneral imaging apparatus 23 are constant, the robot 10 does not have aproblem with such structure. Further, FIG. 4B illustrates a manipulationarm 11 b having a transformable joint member compared to themanipulation arm 11 a illustrated in FIG. 4A. In this case, themanipulation arm 11 b can be transformed in the upward and downwarddirections.

The examples of FIGS. 3 and 4 illustrate the cases where the specialimaging apparatus 21 and the general imaging apparatus 23 are disposedon one manipulation arm 11, but not limited thereto. For example, whenboth the special imaging apparatus 21 and the general imaging apparatus23 or any one of the special imaging apparatus 21 and the generalimaging apparatus 23 are disposed on a plurality of manipulation arms11, variations according to the positional relationship illustrated inFIG. 3 or variations according to the structures of the manipulation arm11 illustrated in FIG. 4 can be used with the same effect.

Variant Example 2 of Robot:

Hereinafter, a description is given of a configuration of the robot 10of variant examples 2-1 and 2-2 with reference to FIGS. 5 and 6.

As to a robot 10 a illustrated in FIG. 5A, the arrangement of thespecial imaging apparatus 21 in the image capture mechanism 20 differsfrom the configuration illustrated in FIG. 2. As to the robot 10 aillustrated in FIG. 5A, the special imaging apparatus 21 is disposed ona support member 13 fixed to the housing 15. With this configuration,the special imaging apparatus 21 always faces the traveling direction ofthe robot 10, with which the operator who operates the robot 10 canoperate the robot 10 easily by viewing the hill-view spherical imageacquired by the special imaging apparatus 21.

As to a robot 10 b illustrated in FIG. 5B, the robot 10 b includes atelescopic member 14 capable of extending and contracting the supportmember 13 of the robot 10 a illustrated in FIG. 5A. As to the robot 10 billustrated in FIG. 5B, the height of the special imaging apparatus 21can be adjusted by extending and contracting the support member 13 usingthe telescopic member 14. With this configuration, the height of thespecial imaging apparatus 21 can be set higher by extending the supportmember 13, with which images of objects surrounding the robot 10 b canbe captured by the special imaging apparatus 21 from the far distance,and the height of the special imaging apparatus 21 can be set lower bycontracting the support member 13, with which the special imagingapparatus 21 can capture images of the ground (foot area) while therobot 10 b moves, and thereby the operation and processing can beperformed flexibly. The support member 13 illustrated in FIGS. 5A and 5Bcan be a pole fixed to the housing 15 or a pedestal fixed to the housing15.

As to a robot 10 c illustrated in FIG. 6A, a mask 27 imitating a humanface is disposed near the special imaging apparatus 21. The mask 27 canrotate about the support member 13 so that the direction of thefull-view spherical image acquired by the special imaging apparatus 21can be seen on the display terminal 50 displaying the full-viewspherical image. With this configuration, the direction that theoperator of the robot 10 is seeing using the full-view spherical imagedisplayed on the display terminal 50 can be informed to persons aroundthe robot 10 by the direction of the mask 27. If this configuration isnot used, persons around the robot 10 feel very uncomfortable andstressful because persons do not know whether they are seen or not bythe robot 10. If the robot 10 c provided with the mask 27 is used,persons around the robot 10 can feel sense of peace of mind because thedirection not facing the mask 27 is not seen by the robot 10.

In FIG. 6, the configuration disposing the mask 27 in the robot 10 c isdescribed, but not limited thereto. For example, the robot 10 c canemploy a configuration disposing a lamp arranged in a circular shape ora spherical shape with respect to the traveling direction of the robot10 c, or a direction display marker, in which the direction that theoperator of the robot 10 is seeing can be informed to persons around therobot 10 c using a light-ON of the lamp or a lighting pattern of thelamp indicating the direction that the operator of the robot 10 isseeing.

Variant Example 3 of Robot:

Hereinafter, a description is given of a configuration of the robot 10of variant example 3 with reference to FIG. 7. As to a robot 10 dillustrated in FIG. 7, the special imaging apparatus 21 and the generalimaging apparatus 23 are disposed on different manipulation arms 11,respectively. With this configuration, the robot 10 d can perform theimaging by the special imaging apparatus 21 from an appropriate positionby transforming the manipulation arm 11 disposing the special imagingapparatus 21, and the robot 10 d can perform the imaging by the generalimaging apparatus 23 at a portion required to be checked in furtherdetail by transforming the manipulation arm 11 disposing the generalimaging apparatus 23. It should be noted that the types of robot of theembodiment can include any type of robots, such as industrial robots,domestic robots, medical robots, service robots, military robots, spacerobots, drones, or the like that can travel in any environment, such asland, air, sea, and underwater depending on application fields of therobots.

Full-View Spherical Image:

Hereinafter, a description is given of an example of a full-viewspherical image acquired by the special imaging apparatus 21 withreference to FIGS. 8 to 12. In this description, it is assumed that aplurality of imaging units, each configured with an image capture deviceand a lens, is provided in the special imaging apparatus 21. Forexample, an image capture unit F is provided at the front side of thespecial imaging apparatus 21, and an image capture unit B is provided atthe rear side of the special imaging apparatus 21.

At first, with reference to FIGS. 8 and 9, a description is given of anoutline of the process until an equirectangular projection image EC anda full-view spherical image CE are created from the images captured bythe special imaging apparatus 21.

FIG. 8A illustrates an example of a hemispherical image (at front side)captured by the special imaging apparatus 21. FIG. 8B illustrates anexample of a hemispherical image (at rear side) captured by the specialimaging apparatus 21, and FIG. 8C illustrates an example of an imageexpressed by the equirectangular projection method (hereinafter,referred to as equirectangular projection image EC). FIG. 9A is aconceptual diagram illustrating a state in which a sphere is coveredwith the equirectangular projection image EC, and FIG. 9B illustrates anexample of the full-view spherical image CE.

As illustrated in FIG. 8A, the image acquired by the image capture unitF provided on the front side of the special imaging apparatus 21 becomesa curved hemispherical image (front side). Further, as illustrated inFIG. 8B, the image acquired by the image capture unit B provided on therear side of the special imaging apparatus 21 becomes a curvedhemispherical image (rear side). Then, the special imaging apparatus 21synthesize the hemispherical image (front side) and the half-sphereimage (rear side) inverted by 180 degrees to create the equirectangularprojection image EC illustrated in FIG. 8C.

Then, the special imaging apparatus 21 uses Open Graphics Library forEmbedded Systems (OpenGL ES) to attach the equirectangular projectionimage by covering the sphere as illustrated in FIG. 9A and creates thefull-view spherical image CE as illustrated in FIG. 9B. Thus, thefull-view spherical image CE is represented as an image in which theequirectangular projection image EC is directed toward the center of thesphere. The OpenGL ES is a graphics library used for visualizing 2D(2-dimensions) data and 3D (3-dimensions) data. Further, the full-viewspherical image CE can be a still image or a video image.

As described above, since the full-view spherical image CE is an imagewhich is attached to cover the spherical surface, humans may be puzzledhen the human sees the image. Therefore, the special imaging apparatus21 can display an image of a part specific region) of the full-viewspherical image CE (hereinafter referred to as a “specific regionimage”) as a planar image having less curvature on a specific display,with which the image not causing puzzlement to the human can bedisplayed. This will be described with reference to FIGS. 10 and 11.

FIG. 10 illustrates a position of a virtual camera IC and a position ofa specific region when a full-view spherical image corresponds to athree-dimensional stereoscopic sphere CS. The virtual camera ICcorresponds to a position of a user viewing the full-view sphericalimage CE displayed as the three-dimensional stereoscopic sphere CS.

Further, FIG. 11A is a perspective view of the virtual camera IC and thethree-dimensional stereoscopic sphere CS of FIG. 10, and FIG. 11Billustrates an example of a specific region image when displayed on thedisplay. Further, FIG. 11A illustrates the full-view spherical image CEof FIG. 9 as the three-dimensional stereoscopic sphere CS. If thegenerated full-view spherical image CE is the stereoscopic sphere CS, asillustrated in FIG. 10, the virtual camera IC is set inside thefull-view spherical sphere image CE. A specific region T in thefull-view spherical image CE is an image capture region of the virtualcamera IC. The specific region T can be specified by specific regioninformation indicating the image capture direction and the angle of viewof the virtual camera IC in a three-dimensional virtual space includingthe full-view spherical image CE.

Then, a specific region image Q indicated in FIG. 11A is displayed as animage of the image capture region of the virtual camera IC on a specificdisplay as illustrated in FIG. 11B. The image illustrated in FIG. 11B isthe specific region image represented by the specific region informationset as the initial setting (default). Hereinafter, a description isgiven using the image capture direction (ea, aa) and the angle of view(α) of the virtual camera IC.

With reference to FIG. 12, the relationship between the specific regioninformation and the image of the specific region I is described. FIG. 12illustrates a relationship between the specific region information andan image of the specific region T. In FIG. 12, “ea” indicates “elevationangle,” “aa” indicates “azimuth angle,” and “α” indicates “angle of view(Angle).” That is, the posture of the virtual camera IC is changed suchthat the point of interest of the virtual camera IC indicated by theimage capture direction (ea, aa) becomes the center point CP of thespecific region T, which is the image capture region of the virtualcamera IC. The specific region image Q is an image of the specificregion T in the full-view spherical image CE. In FIG. 12 “f” denotes adistance from the virtual camera IC to the center point CP. “L” is adistance between a vertex and the center point CP of the specific regionT (2L is a diagonal line of the specific region T). In FIG. 12, thetriangle function represented by the following [Math. 1] is satisfied.

L/f=tan(α2)   [Math. 1]

Hardware Configuration

Hereinafter, a description is given of a hardware configuration of therobot 10, the display terminal 50 and the management server 90 accordingto the embodiment with reference to FIGS. 13 to 15. The hardwareconfiguration illustrated in FIGS. 13 to 15 may have a similarconfiguration in each embodiment, and the components may be added ordeleted if necessary.

Hardware Configuration of Robot:

At first, a hardware configuration of the robot 10 is described withreference to FIG. 13. FIG. 13 illustrates an example of a hardware blockdiagram of the robot 10 according to the embodiment. As to the hardwareblock diagram illustrated in FIG. 13, components can be added or deletedas needed. The robot 10 includes, for example, the control device 30that controls the processing or operation of the robot 10. As describedabove, the control device 30 can be provided inside the housing 15 ofthe robot 10. Further, the control device 30 can be provided outside thehousing 15 of the robot 10 or can be provided as a separate device fromthe robot 10. The control device 30 is an example of an informationprocessing apparatus.

As illustrated in FIG. 13, the control device 30 includes, for example,a central processing Unit (CPU) 301, a read only memory (ROM) 302, arandom access memory (RAM) 303, a hard disk drive (HDD) 304, a networkinterface (I/F) 305, an input-output I/F 306, a media I/F 307, an audioinput/output I/F 308, an external I/F 309, and a bus line 310.

The CPU 301 controls the robot 10 entirely. The CPU 301 is a computingdevice that reads programs or data stored in the ROM 302 or the harddisk (HD) 304 a, and executes the processing to implement the respectivefunctions of the robot 10. The remote-control system 1 a can implementan output control method according to the embodiment by executing someor all of programs according to the embodiment using the CPU 301.

The RAM 303 is a volatile memory used as a working memory and of the CPU301. The ROM 302 is a non-volatile memory that can retain programs ordata even when the power supply is turned off. The HDD 304 controlsreading and writing of various data to the HD 304 a under the control ofthe CPU 301. The HD 304 a stores various kinds of data such as programs.

The network I/F 305 is a communication interface that communicates(connects) with the display terminal 50 via the communication network 9.The network I/F 305 is a communication interface, such as a wired or awireless local area network (LAN). Further, the network I/F 305 can alsoinclude a communication interface, such as 3G (3rd Generation), LTE(Long Term Evolution), 4G (4th Generation), 5G (5th Generation), Zigbee(registered trademark), BLE (Bluetooth (registered trademark) LowEnergy), and millimeter wave radio communication interface.

The input/output I/F 306 is an interface for inputting and outputtingtext, numbers, various instructions, and the like with various externaldevices and the like. The input/output I/F 306 controls display ofvarious information, such as a cursor, a menu, window, text, and imageson a display 306 a, such as liquid crystal display (LCD). The display306 a can be a touch panel display having an input unit. In addition tothe display 306 a, the input/output I/F 306 can be connected to an inputdevice, such as a mouse and a keyboard.

The media 307 controls reading and writing (storing) of data to arecording medium 307 a, such as a universal serial bus (USB) memory, amemory card, an optical disk or a flash memory. The audio input/outputI/F 308 is one or more circuits that processes audio signals input fromthe microphone 308 a and output from the speaker 308 b under the controlof the CPU 301. The external I/F 309 is an interface for connecting thecontrol device 30 with another device.

The bus line 310 is an address bus and a data bus for electricallyconnecting the above components, and transmits address signals, datasignals, various control signals, and the like. The CPU 301, the ROM302, the RAM 303, the HDD 304, the network I/F 30 5, the input/outputI/F 306, the media I/F 307, the audio input/output I/F 308 and theexternal I/F 309 are connected to each other via the bus line 310.

Further, as illustrated in FIG. 13, the control device 30 is connectedto a movement motor 101, an actuator 102, an acceleration/orientationsensor 103, a global positioning system (GPS) receiver 104, a powersupply unit 105, and the above described special imaging apparatus 21and the general imaging apparatus 23 via the external I/F 309.

Based on an instruction from the CPU 301, the movement motor 101 rotatesthe movement mechanism 17 to move the robot 10 on a surface, such asground or floor. The actuator 102 transforms the manipulation arm 11based on an instruction from the CPU 301. The acceleration/orientationsensor 103 includes sensors, such as an electronic magnetic compass fordetecting the geomagnetism, a gyrocompass, and an acceleration sensor.The GPS receiver 104 receives GPS signals from GPS satellites. The powersupply unit 105 is a unit for supplying a power required for the robot10 entirely.

Hardware Configuration Display Terminal:

Hereinafter, a description is given of a hardware configuration of thedisplay terminal 50 with reference to FIG. 14. FIG. 14 illustrates anexample of a hardware block diagram of the display terminal 50 accordingto the embodiment. As illustrated in FIG. 14, the display terminal 50includes, for example, a CPU 501, a ROM 502, a RAM 503, an electricallyerasable programmable read-only memory (EEPROM) 504, an imaging elementa 505, a complementary metal oxide semiconductor (CMOS) sensor 505 a, anacceleration/orientation sensor 506, a media I/F 507, and a GPS receiver508.

The CPU 501 controls the operation of the display terminal 50 entirely.The CPU 501 is a computing device that reads programs and data stored inthe ROM 502 onto the RAM 503, and executes the processing to implementthe respective functions of the display terminal 50. The remote-controlsystem 1 a implements the output control method according to theembodiment of the present invention by executing some or all of theprograms using the CPU 501.

The ROM 502 stores programs used for driving the CPU 501, such as theinitial program loader (IPL) or the like. The RAM 503 is used as aworking memory of the CPU 501. The EEPROM 504 reads or writes variousdata, such as a display terminal program, under the control of the CPU501.

The CMOS sensor 505 a captures an image of an object under the controlof the CPU 501 to obtain image data of the object. The imaging elementI/F 505 is one or more circuits, which controls the driving of the CMOSsensor 505 a. The acceleration/orientation sensor 506 includes varioussensors, such as an electronic magnetic compass for detectinggeomagnetism, a gyrocompass, and an acceleration sensor. The media I/F507 controls reading and writing (storing) of data to a recording medium507 a, such as the flash memory, or the like. The GPS receiver 508receives GPS signals from the GPS satellites.

Further, as illustrated in FIG. 14, the display terminal 50 includes,for example, a long-range communication circuit 510, an antenna 510 afor the long-range communication circuit 510, an audio input/output I/F511, a microphone 511 a, a speaker 511 b, a display 512, an externaldevice connection I/F 513, a short-range communication circuit 514, anantenna 514 a for the short-range communication circuit 514, a touchpanel 515, a timer 516, and a line-of-sight detection device 517.

The long-range communication circuit 510 is one or more circuits, whichcommunicates with another device via the communication network 9. Themicrophone 511 a is, for example, a type of built-in audio collectingunit for inputting audio. The audio input/output I/F 511 is one or morecircuits that processes audio signals input from the microphone 511 aand output from the speaker 511 b under the control of the CPU 501.

The display 512 is a type of display, such as a liquid crystal and anorganic electro-luminescence (OEL) and the like, which can displayimages of objects and various icons. The external device connection I/F513 is an interface for connecting the display terminal 50 with variousexternal devices. The short-range communication circuit 514 is acommunication circuit, such as near field communication (NFC), orBluetooth. The touch panel 515 is a type of input unit for operating thedisplay terminal 50 by the user when a portion of the display 512 ispressed. The timer 516 is a measurement device having a time measurementfunction. The timer 516 can be a software timer implemented by thecomputer. The line-of-sight detection device 517 continuously detectsthe position of the user's line-of-sight as line of sight information.

The line-of-sight detection device 517 includes, for example, an imageprocessing device used for analyzing an image captured by the CMOSsensor 505 a. For example, the line-of-sight detection device 517detects the direction of the line of sight based on the positionalrelationship between the inner corner of the eye and the iris of the eyeby setting the inner corner of the eye as the reference point.

The display terminal 50 further includes a bus line 509. The bus line509 is an address bus and a data bus for electrically connecting each ofthe components, such as the CPU 501.

Hardware Configuration of Management Server: Hereinafter, a descriptionis given of a hardware configuration of the management server 90(control server) with reference to FIG. 15. FIG. 15 illustrates anexample of a hardware block diagram of the management server 90according to the embodiment. The management server 90 is, for example, ageneral computer used as the control server. As illustrated FIG. 15, themanagement server 90 includes, for example, a CPU 901, a ROM 902, a RAM903, an HDD 905, a media I/F 907, a display 908, a network I/F 909, akeyboard 911, a mouse 912, a compact disc-rewritable (CD-RW) drive 914,a timer 915, and a bus line 910. Since the management server 90functions as the control server, the management server 90 can omit somedevices, such as an input device (e.g., keyboard 911, mouse 912) and anoutput device (e.g., display 908).

The CPU 901 controls the operation of the management server 90 entirely.The ROM 902 stores programs to be used for the driving the CPU 901. TheRAM 903 is used as a working memory of the CPU 901. The HDD 905 controlsreading and writing of various data to the HD 904 under the control ofthe CPU 901. The HD 904 stores various kinds of data, such as programs.The media I/F 907 controls data reading and writing (storing) to arecording medium 906, such as a flash memory. The display 908 displaysvarious information, such as a cursor, a menu, a window, text, andimages. The network I/F 909 is an interface for performing datacommunication using the communication network 9. The keyboard 911 is onetype of input unit equipped with a plurality of keys for inputting text,numerals, various instructions, and the like. The mouse 912 is a type ofinput unit for selecting and executing various instructions, selecting aprocess target object, moving a cursor, and the like. The CD-RW drive914 controls reading of various data from the CD-RW 913, which is anexample of a removable recording medium. The timer 915 is a measurementdevice having a time measurement function. The timer 915 can be asoftware timer implemented by the computer.

The management server 90 further includes a bus line 910. The bus line910 is an address bus and a data bus for electrically connecting eachcomponent of the CPU 901 and the like illustrated in FIG. 15.

First Embodiment

Hereinafter, a description is given of a configuration of theremote-control system 1 a according to the first embodiment withreference to FIGS. 16 to 38.

Functional Configuration:

At first, a functional configuration of the remote-control system 1 aaccording to the first embodiment is described with reference to FIG.16. FIGS. 16A and 16B (FIG. 16) illustrate an example of a functionalblock diagram of the remote-control system 1 a according to the firstembodiment.

Functional Configuration of Control Device:

At first, with reference to FIG. 16, a description is given of afunctional configuration of the control device 30 that controls theprocessing or operation of the robot 10. The function implementable bythe control device 30 includes, for example, a transmission/receptionunit 31, an operation input reception unit 32, a display control unit33, a determination unit 34, a state information generation unit 35, animaging instruction unit 36, an image acquisition unit 37, a movementcontrol unit 38, an arm operation control unit 39, a storing/readingunit 41, a position information detection unit 42, and a storage unit3000.

The transmission/reception unit 31 transmits and receives various dataor information to and from other device via the communication network 9.The transmission/reception unit 31 transmits, for example, the full-viewspherical image data 200 or the detailed image data 250 acquired by theimage acquisition unit 37 to the display terminal 50 via thecommunication network 9. The transmission/reception unit 31 is mainlyimplemented by the CPU 301 and the network I/F 305 of FIG. 13. Thetransmission/reception unit 31 is an example of a second transmissionunit in this description.

The operation input reception unit 32 has a function that receives anoperation input to an input unit, such as the display 306 a. Theoperation input reception unit 32 is mainly implemented by the CPU 301and the input/output I/F 306 of FIG. 13.

The display control unit 33 has a function of displaying various screenson the display 306 a. The display control unit 33 is mainly implementedby the CPU 301 and the input/output I/F 306 of FIG. 13.

The determination unit 34 has a function of determining a process or anoperation to be performed by the robot 10 in response to a requestcommand transmitted from the display terminal 50. The determination unit34 is mainly implemented by processing performed by the CPU 301 of FIG.13.

The state information generation unit 35 generates state information 150indicating a state of the robot 10, such as a movement state indicatingwhether or not the robot 10 is moving. The state information generationunit 35 generates and acquires the state information 150 indicating thestate of the robot 10, such as movement state of the robot 10, based ona drive state of the movement mechanism 17 acquired from the movementcontrol unit 38. The details of the state information 150 generated(acquired) by the state information generation unit 35 will be describedlater. The state information generation unit 35 is implemented mainly bythe CPU 301 and the external I/F 309 of FIG. 13. The state informationgeneration unit 35 is an example of a first acquisition unit in thisdescription.

The imaging instruction unit 36 has a function of instructing thespecial imaging apparatus 21 and the general imaging apparatus 23 toperform the imaging process. For example, the imaging instruction unit36 transmits to the special imaging apparatus 21, instructioninformation for instructing the imaging by the special imaging apparatus21. Further, for example, the imaging instruction unit 36 transmits tothe general imaging apparatus 23, instruction information forinstructing the imaging by the general imaging apparatus 23. The imaginginstruction unit 36 is implemented mainly by the CPU 301 and theexternal I/F 309 of FIG. 13.

The image acquisition unit 37 has a function of acquiring the full-viewspherical image acquired by the special imaging apparatus 21 and thedetailed image acquired by the general imaging apparatus 23. :Forexample, the image acquisition unit 37 acquires the full-view sphericalimage data 200 of the full-view spherical image acquired by the specialimaging apparatus 21 by capturing an image of an object, from thespecial imaging apparatus 21. Further, for example, the imageacquisition unit 37 acquires the detailed image data 250 of the detailedimage acquired by the general imaging apparatus 23 by capturing an imageof a part or portion of the object, captured by the special imagingapparatus 21, from the general imaging apparatus 23. The imageacquisition unit 37 is implemented mainly by the CPU 301 and theexternal I/F 309 of FIG. 13. The image acquisition unit 37 is an exampleof a second acquisition unit in this description.

The movement control unit 38 has a function of driving the movementmechanism 17 to control the movement of the robot 10. For example, themovement control unit 38 can move the robot 10 by controlling thedriving of the movement mechanism 17 in response to a request commandtransmitted from the display terminal 50. The movement control unit 38is mainly implemented by the CPU 301 and the external I/F 309 of FIG.13.

The arm operation control unit 39 controls the operation of themanipulation arm 11. For example, the arm operation control unit 39changes the direction or orientation of the manipulation arm 11 bytransforming the manipulation arm 11 based on “ARM” command included inthe request command transmitted from the display terminal 50. The armoperation control unit 39 is mainly implemented by the CPU 301 and theexternal I/F 309 of FIG. 13.

The position information detection unit 42 has a function of acquiringdetection results, such as direction for each bearing (azimuth angle,magnetic north) detected by the acceleration/orientation sensor 103and/or the GPS receiver 104. The detection result of the direction ofeach bearing is positional information indicating the position andorientation of the robot 10 at a specific time. The position informationdetection unit 42 is mainly implemented by the CPU 301 and the externalI/F 309 of FIG. 13. The storing/reading unit 41 has a function ofstoring various data in the storage unit 3000 or reading various kindsof data from the storage unit 3000. The storing/reading unit 41 ismainly implemented by processing performed by the CPU 301 of FIG. 13.The storage unit 3000 is mainly implemented by the ROM 302, the HD 304a, and the recording medium 307 a of FIG. 13.

Further, the storage unit 3000 stores the full-view spherical image data200 and the detailed image data 250 acquired by the image acquisitionunit 37. Further, the storage unit 3000 stores, for example, a commandtable 3001 and an imaging parameter table 3002. The full-view sphericalimage data 200 and the detailed image data 250 stored in the storageunit 3000 can be deleted when a specific time elapses after the imageacquisition unit 37 has acquired the image data, or can be deleted afterthe full-view spherical image data 200 and the detailed image data 250have been transmitted to the display terminal 50.

Command Table:

Hereinafter, a description is given of the details of data stored in thestorage unit 3000 with reference to FIGS. 17A and 17B (FIG. 17). FIG.17A illustrates an example of the command table 3001 according to thefirst embodiment. The command table 3001 illustrated in FIG. 17A is usedto specify the processing or operation to be performed by the robot 10based on the request command transmitted from the display terminal 50.The command table 3001 respectively stores variables and processingcontents corresponding to each one of commands, in association with eachother. The determination unit 34 of the control device 30 specifies theprocessing corresponding to the request command transmitted from thedisplay terminal 50 using the command table 3001.

For example, in the command table 3001, the processing corresponding toa command of “MOVE (variable L, R)” is a process of rotating a leftwheel of the movement mechanism 17 for L° (L degrees) and rotating aright wheel of the movement mechanism 17 for R° (R degrees). Althoughthe robot 10 is assumed to move using two independent left and rightwheels, the same processing can be performed even if the movementmechanism 17 is a foot type or a single wheel as long as the movementmechanism 17 can move into a specific direction

Imaging Parameter Table:

FIG. 117B illustrates an example of the imaging parameter table 3002according to the first embodiment. The imaging parameter table 3002(FIG. 17B) stores parameters setting image quality of the full-viewspherical image captured by the special imaging apparatus 21. In theimaging parameter table 3002, each parameter is stored for each itemdefining the image quality of the full-view spherical image. Forexample, the image quality item includes the frame rate (frame persecond (FPS), update frequency per second) and the resolution(RESOLUTION) of the full-view spherical image. The image quality item isnot limited thereto, but the image quality item may include other itemsrelated to the quality of the full-view spherical image. The imaginginstruction unit 36 of the control device 30 updates (changes) theparameters stored in the imaging parameter table 3002 every time theimage quality of the full-view spherical image to be acquired by thespecial imaging apparatus 21 is changed.

Functional Configuration of Display Terminal:

Hereinafter, a description is given of a functional configuration of thedisplay terminal 50 with reference to FIG. 1613. The functionsimplementable by the display terminal 50 includes, for example, atransmission/reception unit 51, an operation input reception unit 52, adisplay control unit 53, a determination unit 54, a request commandgeneration unit 55, a line-of-sight detection unit 56, a storing/readingunit 57, and a storage unit 5000. The display terminal 50 is installedwith one or more dedicated application programs for performing theremote control of the robot 10. For example, the display terminal 50implements each of the functions by executing the installed applicationprograms using the CPU 501.

The transmission/reception unit 51 transmits and receives various dataor information to and from the other device via the communicationnetwork 9. For example, the transmission/reception unit 51 receives thefull-view spherical image data 200 or the detailed image data 250 fromthe robot 10 (control device 30) via the communication network 9.Further, for example, the transmission/reception unit 51 transmits thestate information 150 indicating the state of the robot 10 from therobot 10 (control device 30) via the communication network 9. Further,for example, the transmission/reception unit 51 transmits a requestcommand generated by the request command generation unit 55 to the robot10 (control device 30) via the communication network 9. Further, forexample, based on the received state information 150, thetransmission/reception unit 51 outputs a request command, which is arequest for imaging to the general imaging apparatus 23 included in therobot 10. The transmission/reception unit 51 is mainly implemented bythe CPU 501 and the long-range communication circuit 510 of FIG. 14. Thetransmission/reception unit 51 is an example of a first acquisition unitin this description. Further, the transmission/reception unit 51 is anexample of a first reception unit in this description. Further, thetransmission/reception unit 51 is an example of a first transmissionunit in this description.

The operation input reception unit 52 has a function of receivingvarious selections or operations input to the display terminal 50. Theoperation input reception unit 52 is mainly implemented by the CPU 501and the touch panel 515 of FIG. 14. It should be noted that the touchpanel 515 may be shared with the display 512. Further, the operationinput reception unit 52 can be implemented using an input unit otherthan the touch panel. The operation input reception unit 52 is anexample of a reception unit in this description.

The display control unit 53 has a function of displaying various imageson the display 512 of the display terminal 50. For example, the displaycontrol unit 53 instructs the display 512 to display the full-viewspherical image data 200 or the detailed image data 250 received by thetransmission/reception unit 51. Further, based on the state information150 received (acquired) by the transmission/reception unit 51, thedisplay control unit 53 switches the image data displayed on the display512 between the full-view spherical image data 200 and the detailedimage data 250. That is, the display control unit 53 outputs thefull-view spherical image data 200 or the detailed image data 250 basedon the state information 150 received (acquired) by thetransmission/reception unit 51. The display control unit 53 is mainlyimplemented by the CPU 501 and the display 512 illustrated in FIG. 14.The display control unit 53 is an example of a display control unit inthis description. Further, the display 512 is an example of a displayunit in this description.

The determination unit 54 has a function of determining a specificprocess to be requested to the robot 10. For example, the determinationunit 54 determines the specific process to be requested to the robot 10based on an operation input received by the operation input receptionunit 52. Further, the determination unit 54 determines the specificprocess to be requested to the robot 10 based on the state information150 received (acquired) by the transmission/reception unit 51. Thedetermination unit 54 is mainly implemented by processing performed bythe CPU 501 of FIG. 14.

The request command generation unit 55 generates a request command to berequested to by the robot 10, which is a request for executing aspecific process by the robot 10. For example, the request commandgeneration unit 55 generates the request command, such as a request forimaging to be requested to the general imaging apparatus 23 included inthe robot 10. Hereinafter, the request for imaging to be requested tothe general imaging apparatus 23 is referred to as the imaging request.The request command generation unit 55 is mainly implemented byprocessing performed by the CPU 501 of FIG. 14.

The line-of-sight detection unit 56 detects the line of sight of anoperator who operates the robot 10 using the display terminal 50. Theline-of-sight detection unit 56 is mainly implemented by the CPU 501 andthe line-of-sight detection device 517 of in FIG. 14.

The storing/reading unit 57 has a function of storing various data inthe storage unit 5000 or reading various kinds of data from the storageunit 5000. The storing/reading unit 57 is mainly implemented byprocessing performed by the CPU 501 of FIG. 14. The storage unit 5000 ismainly implemented by the ROM 502, the EEPROM 504, and the recordingmedium 507 a of FIG. 14.

Further, the storage unit 5000 stores the full-view spherical image data200 and the detailed image data 250 received by thetransmission/reception unit 51. The storage unit 5000 further stores,for example, a state management table 5001 (state control table), acondition table 5002, and a user command table 5003. Further, thefull-view spherical image data 200 and the detailed image data 250stored in the storage unit 5000 can be deleted when a given time elapsesafter receiving the data by the transmission/reception unit 51, or thefull-view spherical image data 200 and the detailed image data 250 canbe deleted based on a user's deletion instruction received by theoperation input reception unit 52.

State Management Table:

Hereinafter, a description is given of contents of data stored in thestorage unit 5000. FIG. 18 is an example of the state management table5001 (state control table) according to the first embodiment.Specifically, the current state of the robot 10 can be stored in thestate management table 5001 as illustrated in FIG. 18. For example, thestate management table 5001 stores a value indicating the currentmovement state of the robot 10 for the traveling direction and thetraveling velocity of the robot 10. The traveling direction of the robot10 is specified or defined by the horizontal angle (H_ANGLE) and thevertical angle (V_ANGLE). Each time the robot 10 moves, the displayterminal 50 updates (changes) values of each item included in the statemanagement table 5001. Specifically, the storing/reading unit 57 of thedisplay terminal 50 updates the contents of the state management table5001 based on the state information 150 (see FIG. 28) received by thetransmission/reception unit 51.

Condition Table:

FIG. 19 is an example of the condition table 5002 according to the firstembodiment. The condition table 5002 (FIG. 19) is used to specify thecontents of processing or operation to be requested to the robot 10based on the state information 150 received by thetransmission/reception unit 51. The condition table 5002 stores, forexample, conditions of the movement state of the robot 10 in associationwith contents of processing and contents of command, to be transmittedto the robot 10. For example, if the traveling velocity of the robot 10becomes greater than 5.0 km (“SPEED>5 km/h”), the display terminal 50specifies the processing content of “lowering the frame rate to 3” asthe content of the processing and specifies “FPS(3)” as a command to betransmitted to the robot 10. In the condition table 5002, “RESOLUTION”is a command related to the resolution of the full-view spherical image,and “ZOOM” is a command related to a display range (output range) of thefull-view spherical image. The conditions and/or the processing contentsincluded in the condition table 5002 are not limited thereto, but can bemodified, changed, added, and deleted appropriately by an operator orthe like who operates the robot 10 using the display terminal 50.

User Command Table:

FIG. 20 is an example of the user command table 5003 according to thefirst embodiment. The user command table 5003 (FIG. 20) is used tospecify the contents of processing or operation to be requested to therobot 10 based on an operation input received by the operation inputreception unit 52. The user command table 5003 stores input commandscorresponding to operation inputs received by the operation inputreception unit 52 in association with respective correspondingprocessing content and corresponding type of processing.

Functional Configuration of Management Server:

Hereinafter, a description is given of a functional configuration of themanagement server 90 (control server) with reference to FIG. 16. Thefunctions implementable by the management server 90 include, forexample, a transmission/reception unit 91, an authentication unit 92, adetermination unit 93, a creation unit 94, a storing/reading unit 95,and a storage unit 9000. To be described later, the storage unit 9000includes an authentication management database (DB) 9001, a terminalmanagement DB 9002, a destination list management DB 9003, and a sessionmanagement DB 9004.

The transmission/reception unit 91 transmits and receives various dataor information to and from other device via the communication network 9.The transmission/reception unit 91 is mainly implemented by the CPU 901and the network I/F 909 of FIG. 15.

The authentication unit 92 performs authentication of a login requestingsource (e.g., user, apparatus) based on a login request received by thetransmission/reception unit 91. For example, the authentication unit 92searches the authentication management DB 9001 of the storage unit 9000using a terminal identification (ID) and a password included in thelogin request received by the transmission/reception unit 91 as a searchkey. Then, the authentication unit 92 performs the authentication of aterminal by determining whether or not the same set of terminal ID andpassword are stored in the authentication management DB 9001. Theauthentication unit 92 is mainly implemented by processing performed bythe CPU 901 of FIG. 15.

The determination unit 93 has a function of determining whether theterminal ID of the display terminal 50 is managed in a sessionmanagement table to be described later. The determination unit 93 ismainly implemented by processing performed by the CPU 901 of FIG. 15.

The creation unit 94 has a function of creating a session identification(ID) to be used for communication. The creation unit 94 is mainlyimplemented by processing performed by the CPU 901 of FIG. 15.

The storing/reading unit 95 stores various data in the storage unit 9000or reads various kinds of data from the storage unit 9000. Thestoring/reading unit 95 is mainly implemented by processing performed bythe CPU 901 of FIG. 15.

The storage unit 9000 is mainly implemented by the ROM 902, the HD 904,or the recording medium 906 of in FIG. 15. Further, the storage unit9000 stores destination list frame data used for a destination listscreen 500 illustrated in FIG. 24 to be described later. The destinationlist frame data does not include destination list content information,such as icons, “rA01,” and “robot 10A-1” indicated in FIG. 24.

Authentication Management Table:

The storage unit 9000 stores the authentication management DB 9001(authentication control DB) including an authentication management table(authentication control table) illustrated in FIG. 21A. In theauthentication management table, information associated with eachpassword is managed for the respective terminal IDs of all of thedisplay terminals 50 managed or controlled by the management server 90.For example, in the authentication management table illustrated in FIG.21A, the terminal ID of the display terminal 50A is “o01” and thecorresponding password is “aaaa.”

Terminal Management Table:

The storage unit 9000 stores the terminal management DB 9002 (terminalcontrol DB) including a terminal management table (terminal controltable) illustrated in FIG. 21B. The terminal management table storesinformation related to the terminal ID of each terminal (e.g., robot 10,display terminal 50), such as terminal name of each terminal, internetprotocol (IP) address of each terminal, operation state informationindicating the current operation state of each terminal, and site nameindicating a site where the robot 10 is located in a case that theterminal is the robot 10, which are stored in association with eachother.

For example, in the terminal management table illustrated in FIG. 21B,the display terminal 50 having the terminal ID of “001” has the terminalname of “display terminal 50A,” the IP address of “1.2.1.3” and theoperation state of “online (can communicate).” Further, the robot 10having the terminal ID of “rA01” has the terminal name of “robot 10A-1,”the IP address of “1.3.2.3,” the operation state of “online (cancommunicate)” and the site name of “site A.”

Destination List Management Table:

Further, the storage unit 9000 stores the destination list management DB9003 (destination list control DB) including a destination listmanagement table (destination list control table) illustrated in FIG.22A. Specifically, the destination list management table storesinformation related to the terminal ID of the display terminal 50 andthe terminal ID of the robot 10 (i.e., destination candidate, which arestored in association with each other. For example, the display terminal50 is used as a starting terminal for requesting a start ofcommunication for performing the remote control of the robot 10, and therobot 10 is registered as the destination candidate to be operated underthe remote control performed from the display terminal 50.

For example, in the destination list management table illustrated inFIG. 22A, the destination candidate that the starting terminal (i.e.,display terminal 50A) having the terminal ID of “o01” can request astart of communication includes the robot 10A-1 having the terminal IDof “rA01,” the robot 10A-2 having the terminal ID of “rA02,” and therobot 10C-1 having the terminal ID of “rC01.” The terminal ID of therobot 10 settable as the destination candidate can be updated by addingor deleting the terminal ID of the robot 10 based on a request of addingor deleting the terminal ID of the robot 10 to the management server 90from any starting terminal (e.g., display terminal 50).

Session Management Table:

Further, the storage unit 900( )stores the session management DB 9004(session control DB) including a session management table (sessioncontrol table) illustrated in FIG. 22B. The session management tableincludes the session ID and the terminal ID of the display terminal 50and the robot 10, which are stored in association with each other. Thesession ID identifies a session used for communication between the robot10 and the display terminal 50. The terminals ID of the robot 10 and theterminal ID of the display terminal 50 using a session, identified bythe session ID, are associated with the session ID. For example, in thesession management table illustrated in FIG. 22B, the terminal using thesession performed by using the session ID of “se1” includes the displayterminal 50 a having the terminal ID of “o01,” the robot 10A-2 havingthe terminal ID of “rA02,” and the robot 10C-1 having the terminal ID of“rC01,”

Processing and Operation in First Embodiment:

Hereinafter, a description is given of operation and processing of theremote-control system 1 a according to the first embodiment withreference to FIGS. 23 to 38. In the following description, theprocessing performed by the control device 30 included in the robot 10will be described as the processing performed by the robot 10.

Processing of Session Establishment:

At first, a communication session establishment process between therobot 10 and the display terminal 50 is described with reference toFIGS. 23 to 25. FIG. 23 is an example of a sequence diagram illustratinga preparatory stage for starting data transmission and reception betweenthe robot 10 and the display terminal 50. Hereinafter, a description isgiven of the transmission/reception processing for each managementinformation in the preparatory stage before starting the datatransmission/reception between the display terminal 50A, used as thestarting terminal, and the robot 10A-1, used as the destinationterminal.

At first, in step S11, the transmission/reception unit 51 of the displayterminal 50A transmits a login request to the management server 90 viathe communication network 9. Specifically, when a user of the displayterminal 50A turns on the power switch (ON) of the display terminal 50A,the power supply is turned ON. In response to the turning on of thepower supply, the transmission/reception unit 51 of the display terminal50A transmits a login request to the management server 90 via thecommunication network 9. With this configuration, thetransmission/reception unit 91 of the management server 90 receives thelogin request transmitted from the display terminal 50A.

The login request includes, for example, a terminal ID identifying thestarting terminal such as the display terminal 50A and a password. Theterminal ID and password are data, read by the storing/reading unit 57from the storage unit 5000, and transmitted to thetransmission/reception unit 51. The terminal ID and password are notlimited thereto. For example, the terminal ID and password input by auser using the input unit such as the touch panel 515 can betransmitted. Further, the terminal ID and password read out from astorage medium such as a subscriber identity module (SIM) card or asecure digital (SD) card connected to the display terminal 50A can betransmitted.

Further, when the login request is transmitted from the display terminal50A to the management server 90, the management server 90, which is areceiving side of the login request, can acquire the IP address of thedisplay terminal 50A, which is a transmitting side of the login request.Further, the start of login request does not necessarily at the time ofturning on the power switch (ON). For example, the login request can betransmitted in response to an input to the input unit such as the touchpanel 515 by a user.

In step S12, the authentication unit 92 of the management server 90searches the authentication management table (FIG. 21A) in the storageunit 9000 using the terminal ID and password included in the loginrequest received via the transmission/reception unit 91 as a search key,and determines whether the authentication management DB 9001 stores thesame terminal ID and the same password to perform the authentication.Hereinafter, a description is given of a case when the authenticationunit 92 determines that the display terminal 50A is a terminal having avalid use authentication.

If the authentication unit 92 of the management server 90 determinesthat the login request is transmitted from the starting terminal havingthe valid use authentication based on the same terminal ID and the samepassword stored in the authentication management DB 9001 in step S12, instep S13, the storing/reading unit 95 reads out the destination listframe data from the storage unit 9000.

In step S14, the transmission/reception unit 91 transmits authenticationresult information indicating the authentication result determined bythe authentication unit 92, to the display terminal 50 that hastransmitted the login request, via the communication network 9. Then,the transmission/reception unit 51 of the display terminal 50 receivesthe authentication result information. The authentication resultinformation includes the destination list frame data read out in stepS13.

In step S15, the storing/reading unit 57 of the display terminal 50Astores the destination list frame data, received in step S14, in thestorage unit 5000.

If the transmission/reception unit 51 of the display terminal 50Areceives the authentication result information indicating theauthentication result determining that the terminal 50A has the validuse authentication, in step S16, the transmission/reception unit 51 ofthe display terminal 50A requests contents of the destination list tothe management server 90 via the communication network 9. Then, thetransmission/reception unit 91 of the management server 90 receives arequest of the contents of destination list. The request includes theterminal ID of the display terminal 50A.

In step S17, the storing/reading unit 95 of the management server 90uses the terminal ID of “o01” of the display terminal 50A, received instep S16, as a search key to search the destination list managementtable (FIG. 22A) to read out the terminal ID of one or more of thecorresponding destination candidates.

In step S18, the storing/reading unit 95 uses the terminal ID, read instep S17, as a search key to search the terminal management table (FIG.21B) to read out the terminal name, operation state information, andsite name of the corresponding destination candidate.

In step S19, the transmission/reception unit 91 of the management server90 transmits the destination list content information to the displayterminal 50A via the communication network 9. Then, thetransmission/reception unit 51 of the display terminal 50A receives thedestination list content information. The destination list contentinformation includes, for example, the terminal ID of destinationcandidate, the terminal name of destination candidate, the operationstate information of destination candidate, and the site name ofdestination candidate, read out in steps S17 and S18.

In step S20, the display control unit 53 of the display terminal 50Ainstructs the display 512 to display the destination list screen 500FIG. 24), which is generated from the destination list frame data storedin the storage unit 5000 in step S15 and the destination list contentinformation received in step S19.

The display 512 displays the destination list screen 500 as illustratedin FIG. 24. Specifically, the destination list screen 500 displays anicon indicating the operation state of the destination candidateterminal (e.g., robot 10), a terminal ID of the destination candidateterminal, a destination name of the destination candidate terminal, anda site name where the destination candidate terminal is located for eachof the destination candidate terminals. The “terminal name” transmittedfrom the management server 90 in step S19 can be displayed as“destination name” in the destination list screen 500 illustrated inFIG. 24.

Hereinafter, a description is given of a process from selecting thedestination candidate to starting the transmission/reception of imagedata using the display terminal 50 with reference to FIG. 25. FIG. 25 isan example of a sequence diagram illustrating processing from selectingthe destination candidate (e.g., terminal, apparatus) to starting thetransmission and reception of image data.

At first, in step S31, the operation input reception unit 52 of thedisplay terminal 50A receives a selection of destination candidate(e.g., robot 10A-1) on the destination list screen 500 (FIG. 24) from auser.

In step S32, the transmission/reception unit 51 of the display terminal50A transmits a start request indicating that the display terminal 50Ais ready to start the transmission/reception of image data, to themanagement server 90. Then, the transmission/reception unit 91 of themanagement server 90 receives the start request from the displayterminal 50A. The start request includes, for example, the terminal IDof the display terminal 50A and the terminal ID of the selecteddestination candidate.

In step S33, the determination unit 93 of the management server 90determines whether the terminal ID of the display terminal 50A, receivedin step S32, is stored in the session management table (FIG. 22B) in thesession management DB 9004. Hereinafter, a description is given of acase that the terminal ID of the destination candidate terminal (i.e.,robot 10A-1) is not stored in the session management table in thesession management DB 9004.

If the terminal ID of the destination candidate is not stored in thesession management table, in step S34, the creation unit 94 of themanagement server 90 creates a new session ID.

In step S35, the storing/reading unit 95 additionally stores a newrecord in the session management table (FIG. 22B), in which the newsession ID created in step S34, and the terminal ID of the displayterminal 50A and the terminal ID of the destination candidate terminal,received in step S32, are associated with each other and stored as thenew record. In this case, as illustrated in FIG. 22B, the session ID of“se3” and the terminal IDs of “o01” and “rA01” associated with eachother are added and stored as the new record.

In step S36, the transmission/reception unit 91 of the management server90 transmits a session start instruction including the session IDcreated in step S34 to the display terminal 50A. Then, thetransmission/reception unit 51 of the display terminal 50A receives thesession start instruction from the management server 90.

In step S37, the storing/reading unit 95 of the management server 90searches the terminal management table (FIG. 21B) using the terminal IDof the destination candidate terminal (i.e., robot 10A-1), received instep S32, to read out the corresponding IP address of the destinationcandidate terminal from the terminal management table.

In step S38, the transmission/reception unit 91 of the management server90 transmits a session start instruction including the session ID,created in step S34, to the IP address read out in step S37. Then, thetransmission/reception unit 31 of the destination terminal (i.e., robot10A-1) receives the session start instruction from the management server90.

Then, each of the starting terminal (i.e., display terminal 50A) and thedestination terminal (i.e., robot 10A-1) establishes a communicationsession with the management server 90 in steps S39-1 and S39-2.

Transmission and Display of Image Data:

Hereinafter, a description is given of data transmitted from the robot10 to the display terminal 50 and controlling the processing oroperation of the robot 10 using the display terminal 50 establishing thecommunication session with the management server 90. FIG. 26 is anexample of a sequence diagram illustrating a transmission process ofvarious data from the robot 10 to the display terminal 50 in theremote-control system 1 a according to the first embodiment.

In steps S51-1 and S51-2, the transmission/reception unit 31 of therobot 10 transmits the full-view spherical image data 200 acquired bythe special imaging apparatus 21 to the display terminal 50 using thecommunication session established with the management server 90. Then,the transmission/reception unit 51 of the display terminal 50 receivesthe full-view spherical image data 200 from the robot 10 via themanagement server 90.

In this case, the robot 10 starts the image capturing using the specialimaging apparatus 21 based on an imaging instruction transmitted fromthe imaging instruction unit 36 to the special imaging apparatus 21 as atrigger. The image acquisition unit 37 of the robot 10 acquires thefull-view spherical image data 200, which is the full-view sphericalimage acquired by the special imaging apparatus 21, from the specialimaging apparatus 21. Then, the transmission/reception unit 31 of therobot 10 transmits the full-view spherical image data 200 acquired bythe image acquisition unit 37 to the display terminal 50 via themanagement server 90.

In step S52, the display control unit 53 of the display terminal 50instructs the display 512 to display a display screen 600 a (FIG. 27)for displaying the full-view spherical image data 200 received by thetransmission/reception unit 51. With this configuration, an operator whoremotely operates the robot 10 using the display terminal 50 can confirmthe status of the site where the robot 10 is located by viewing thedisplay screen 600 a displaying the full-view spherical image data 200.

Since the operator of the robot 10 remotely operates the robot 10 byviewing the full-view spherical image data 200 being displayed on thedisplay terminal 50, the real-time property is required for thefull-view spherical image data 200 being displayed on the displayterminal 50. Therefore, the transmission/reception unit 31 of the robot10 continuously or constantly transmits the full-view spherical imagedata 200 acquired by the special imaging apparatus 21 to the displayterminal 50 via the management server 90. Therefore, it is preferablethat the full-view spherical image data 200 acquired by the specialimaging apparatus 21 is video image data. With this configuration, theoperator who operates the robot 10 by using the display terminal 50 canremotely control the robot 10 by checking the video image data of thefull-view spherical image transmitted from the robot 10 and reproducedby the streaming play on the display terminal 50, with which theoperator can check the surrounding of the robot 10 extensively or widerrange without changing the direction of the special imaging apparatus 21and/or the robot 10.

FIG. 27 illustrates an example of the display screen 600 a displayed onthe display terminal 50 according to the first embodiment. In thedisplay terminal 50, the display control unit 53 displays the displayscreen 600 a on the display 512 as illustrated in FIG. 27. The displayscreen 600 a includes, for example, a first display field 610, a seconddisplay field 630, a “home” button 601, a “shoot” button 603, a movementinstruction key 605, a speed bar 607, and a zoom bar 613 as illustratedin FIG. 27.

The first display field 610 displays the full-view spherical image data200 transmitted from the robot 10. The first display field 610 is alsoreferred to as the full-view spherical image display field. The seconddisplay field 630 displays the detailed image data 250 transmitted fromthe robot 10. The second display field 630 is also referred to as thedetailed image data display field.

The “home” button 601 is pressed when the first image capture directionof the full-view spherical image data 200 displayed in the first displayfield 610 is changed (reset) to the current traveling direction of therobot 10. The “shoot” button 603 is pressed when transmitting theimaging requesting to the general imaging apparatus 23 included in therobot 10.

The movement instruction key 605 is pressed when requesting the movementof the robot 10 in the horizontal direction (forward, rearward, rightrotation, left rotation). The speed bar 607 displays a movement speedindicating the current state of the movement speed of the robot 10. Thezoom bar 613 indicates the current zoom level of the full-view sphericalimage data 200 displayed in the first display field 610.

The operator who remotely controls the robot 10 using the displayterminal 50 can select the respective tabs of the first display field610 and the second display field 630 set at the top left on each displayfield to switch the image display between the full-view spherical imageand the detailed image. In the first display field 610, a part orportion of the full-view spherical image data 200 transmitted from therobot 10 (e.g., a specific region image Q in FIG. 11) is displayed. Theoperator moves a mouse pointer 620 to perform a specific input operationwithin the first display field 610 displaying the full-view sphericalimage data 200 to change the first image capture direction and/or thezoom level of the full-view spherical image data 200.

Further, if the operator sees the full-view spherical image data 200displayed in the first display field 610 and then wants to confirm themore detailed image, the operator presses the “shoot” button 603. Inthis case, in response to receiving the pressing of the “shoot” button603 at the operation input reception unit 52, the request commandgeneration unit 55 of the display terminal 50 generates a requestcommand requesting the imaging in the direction same as the first imagecapture direction of the full-view spherical image data displayed on thefirst display field 610, to the general imaging apparatus 23. Thedetailed image data 250, which is the detailed image acquired by thegeneral imaging apparatus 23, is acquired by the image acquisition unit37 of the robot 10 and then transmitted to the display terminal 50 fromthe transmission/reception unit 31 via the management server 90. Then,the display terminal 50 instructs the display control unit 53 to displaythe detailed image data 250 received by the transmission/reception unit51 in the second display field 630.

In an example case of FIG. 27, the movement of the robot 10 is remotelycontrolled by receiving an operation input to the movement instructionkey 605 displayed on the display screen 600 a, but not limited thereto.For example, the movement operation of the robot 10 can be performedusing a dedicated controller, such as a game pad equipped with akeyboard or a joy stick.

Further, the display terminal 50 can employ a method of displaying, onthe display screen 600 a, a location name of the movement destinationrequested to the robot 10 and a list of persons to be met, with which auser can select or input an address of the requested movementdestination. In this case, the display terminal 50 transmits a requestcommand for moving the robot 10 to the input specified destination tothe robot 10, and then the robot 10 autonomously moves to thedestination specified by the request command.

The description returns to FIG. 26. In step S53, the robot 10 moveswithin the site based on the request command transmitted from thedisplay terminal 50 to be described later. In this case, the movementcontrol unit 38 of the robot 10 controls driving of the movementmechanism 17 based on a request command transmitted from the displayterminal 50 to be described later.

In step S54, the state information generation unit 35 of the robot 10generates the state information 150 indicating the movement state of therobot 10 based on the drive state of the movement mechanism 17 acquiredfrom the movement control unit 38.

FIG. 28 illustrates an example of the state information 150 according tothe first embodiment. As illustrated in FIG. 28, the state information150 includes, for example, information corresponding to items, such asthe traveling direction (horizontal angle and vertical angle) and thetraveling velocity of the robot 10. The state information 150 includes,for example, a variable name corresponding to each item and a numericalvalue corresponding to the variable name. For example, in a case of thestate information 150 illustrated in FIG. 28, the robot 10 is moving inthe traveling direction defined by the horizontal angle of “37.2°” andthe vertical angle of “45.3°” with the traveling velocity of “3.0 km/h.”In this example case, the state information 150 includes information ofthe traveling direction (horizontal angle and vertical angle) and thetraveling velocity of the robot 10 as the movement state of the robot10, but not limited thereto. For example, the state information 150 caninclude information on the traveling distance of the robot 10.

Then, in steps S55-1 and S55-2, the transmission/reception unit 31 ofthe robot 10 transmits the state information 150 generated by the stateinformation generation unit 35 to the display terminal 50 using thecommunication session established with the management server 90. Then,the transmission/reception unit 51 of the display terminal 50 receives(acquires) the state information 150 from the robot 10 via themanagement server 90.

In step S56, the storing/reading unit 57 of the display terminal 50stores the state information 150 received (acquired) by thetransmission/reception unit 51 in the state management table 5001 (FIG.18) stored in the storage unit 5000. Specifically, the storing/readingunit 57 of the display terminal 50 stores the respective numericalvalues included in the state information 150, received (acquired) by thetransmission/reception unit 51, in the corresponding variable name ofthe state management table 5001 to update the numerical values of eachitem included in the state management table 5001.

Then, in step S57, the display control unit 53 of the display terminal50 displays the movement state of the robot 10 on the display screen asillustrated in FIG. 29.

FIG. 29A is an example of a screen displayed as a display screen 600 bwhen the robot 10 is moving in a forward direction. The display screen600 b indicates that the robot 10 is moving in the forward direction bychanging a color of a forward key in the movement instruction key 605.The display control unit 53 of the display terminal 50 updates thedisplay of the movement instruction key 605 based on the numericalvalues corresponding to the variable names of “V_ANGLE” and “H_ANGLE,”which are included in the state information 150 received by thetransmission/reception unit 51. Further, the display control unit 53 canbe configured to update the display of the movement instruction key 605by receiving the operation input to the movement instruction keg 605.

Further, the display screen 600 b indicates the traveling velocity ofthe robot 10 using a ratio of a black portion in the speed bar 607. Thedisplay control unit 53 of the display terminal 50 updates the displayof the speed bar 607 based on the numerical value corresponding to thevariable name of “SPEED” included in the state information 150 receivedby the transmission/reception unit 51.

FIG. 29B is another example of a screen displayed as a display screen600 c when the robot 10 is moving in a forward direction. In the displayscreen 600 c illustrated in FIG. 29B, a display range (output range) ofthe full-view spherical image displayed on the first display field 610is different from the display screen 600 b illustrated in FIG. 29A.Since the display screen 600 c displays a zoom-out full-view sphericalimage on the first display field 610 (in a zoom-out state), the bar inthe zoom bar 613 is moved to the downward. The zoom bar 613 indicatesthe zoom level of the full-view spherical image displayed on the firstdisplay field 610 by setting a bar indicator along the top and bottom ofthe zoom bar 613.

Further, since the display screen 600 c displays the full-view sphericalimage in the zoom-out state, a head 615 of the robot 10 is alsodisplayed. In this case, the operator who operates the robot 10 usingthe display terminal 50 can recognize the approximate positionalrelation and distance between the robot 10 and the periphery of therobot 10. Conventionally, the displaying of the display screen 600 crequires to dispose another camera at the rear side of the robot 10. Bycontrast, the display terminal 50 can display the display screen 600 cusing the image captured by the special imaging apparatus 21 (oneimaging apparatus)and providing the zoom function of the displayedfull-view spherical image.

FIG. 30 is another example of a screen displayed as a display screen 600d when the robot 10 is moving in a forward direction. The display screen600 d in FIG. 30 indicates the traveling direction of the robot 10 inthe first display field 610. In cases of the display screen 600 billustrated in FIG. 29A and the display screen 600 c illustrated in FIG.29B, when the operator presses the “home” button 601, the first imagecapture direction of the full-view spherical image displayed on thefirst display field 610 can be changed to the current travelingdirection of the direction of the robot 10, with which the operator canconfirm the traveling direction of the robot 10.

On the other hand, in the display screen 600 c, a traveling directionicon 617 indicating the traveling direction of the robot 10 issuperimposed and displayed on the full-view spherical image displayed onthe first display field 610. With this configuration, the operator canconfirm the traveling direction of the robot 10 by viewing the displayscreen 600 c regardless of the first image capture direction of thefull-view spherical image.

In the display screen 600 d, the length of the traveling direction icon617 can be changed depending on the traveling velocity of the robot 10.Further, when the robot 10 is stopped, the traveling direction of therobot 10 just before the robot 10 is stopped can be displayed using thetraveling direction icon 617 in the display screen 600 d. In this case,it is preferable that the display style of the traveling direction icon617 when the robot 10 is stopped differs from the display style of thetraveling direction icon 617 when the robot 10 is moving, such as adotted line or a changed color can be used as the display style of thetraveling direction icon 617 when the robot 10 is stopped. With thisconfiguration, the remote-control system 1 a can display the full-viewspherical image data 200 transmitted from the robot 10 (control device30) on the display terminal 50, with which the operator who performs theremote control of the robot 10 using the display terminal 50 canrecognize information around the robot 10 more accurately. Further, thedisplay terminal 50 can display information indicating the movementstate of the robot 10 together with the full-view spherical image data200, with which the display terminal 50 can notify the operator whoperforms the remote control of the robot 10 using the display terminal50, the information around the robot 10 and the movement state of therobot 10.

The state information generation unit 35 of the robot 10 can beconfigured to generate the state information 150 if a change occurs inthe drive state of the movement mechanism 17 acquiring from the movementcontrol unit 38. Further, the state information generation unit 35 canbe also configured to generate the state information 150 with a pre-setgiven frequency. Further, the state information 150 generated by thestate information generation unit 35 can be configured to be retained,and the transmission/reception unit 31 of the robot 10 can be configuredto transmit the retained state information 150 with a pre-set givenfrequency. In this case, the remote-control system 1 a can reducedifficulty of seeing the display screen caused by the frequent change ofthe movement state of the robot 10 displayed on the display terminal 50.

Remote Control of Robot:

Hereinafter, a description is given of the remote control of the robot10 performed by using a display screen displayed on the display terminal50. At first, an example of an operation control of the robot 10 basedon the state information 150 transmitted from the robot 10 is describedwith reference to FIG. 31. FIG. 31 is an example of a flowchartillustrating a robot control process based on the movement state of therobot 10 using the display terminal 50 according to the firstembodiment. In step S61 a, if the transmission/reception unit 51 of thedisplay terminal 50 receives (acquires) the state information 150transmitted from the robot 10 (S61 a: YES), the sequence proceeds tostep S62 a. On the other hand, if the transmission/reception unit 51does not receive (acquire) the state information 150 (S61 a: NO), thesequence repeats step S61 a.

In step S62 a, the storing/reading unit 57 of the display terminal 50reads out the condition table 5002 (FIG. 19) stored in the storage unit5000.

In step S63 a, the determination unit 54 of the display terminal 50searches the condition table 5002 read out by the storing/reading, unit57, and if there is any variable that satisfies the condition includedin the condition table 5002 among the variables included in the stateinformation 150 received by the transmission/reception unit 51 (S63 a:YES), the determination unit 54 proceeds the sequence to step S64 a. Forexample, when the numerical value corresponding to the variable name of“SPEED” included in the state information 150 is “3.0 km/h,” thedetermination unit 54 determines that “SPEED≤5.0 km/h,” “SPEED>1.0 km/h”and “SPEED>2.0 km/h” satisfy the condition among the items included inthe condition table 5002 (FIG. 19). The variable may be also referred toas the variable item, or item in this description.

On the other hand, in step S63 a, if the determination unit 54determines that the variables included in the state information 150received by the transmission/reception unit 51 do not satisfy thecondition included in the condition table 5002 (S63 a: NO), thedetermination unit 54 terminates or end the sequence.

In step S64 a, the request command generation unit 55 of the displayterminal 50 specifies the process corresponding to the variable itemdetermined in step S63 to generate a request command. For example, therequest command generation unit 55 specifies a specific processcorresponding to each of “SPEED≤5.0 km/h,” “SPEED>1.0 km/h” and“SPEED>2.0 km/h” satisfying the condition included in the conditiontable 5002 (FIG. 19). Then, the request command generation unit 55generates a request command using a command corresponding to thespecified processing included in the condition table 5002. In this case,the commands included in the request command are “FPS(30),”“RESOLUTION(50%)” and “ZOOM=10.” It should be noted that the requestcommand can include all commands corresponding to the items satisfyingthe conditions, or one or more commands corresponding to the itemsselected (extracted) from all of the items satisfying the conditions.

In step S65 a, the transmission/reception unit 51 of the displayterminal 50 transmits the request command generated by the requestcommand generation unit 55 to the robot 10 via the communication network9. Then, the transmission/reception unit 31 of the robot 10 receives therequest command corresponding to the current movement state of the robot10 from the display terminal 50. When the robot 10 receives the requestcommand, the robot 10 executes a command processing (see FIG. 33) whichwill be described later.

As above described, if the display terminal 50 requests to change theimage quality of the full-view spherical image of the full-viewspherical image data 200 acquired by the special imaging apparatus 21based on the movement state of the robot 10, the image quality of thefull-view spherical image displayed on the display terminal 50 can bechanged. For example, when the robot 10 is moving, the display terminal50 can transmit a request for reducing the image quality of thefull-view spherical image to the robot 10. Specifically, the displayterminal 50 transmits a request for reducing the resolution of thefull-view spherical image and/or reducing the frame rate (updatefrequency) to the robot 10. With this configuration, by reducing theresolution and/or frame rate (update frequency) of the full-viewspherical image displayed on the display terminal 50, the remote-controlsystem 1 a can reduce discomfort of the operator viewing the full-viewspherical image data 200 displayed on the display terminal 50 whileoperating the robot 10, such as feeling of intoxication or the like.

Further, when performing the remote-control operation of the robot 10using the display terminal 50, the operator performs the operation inputwhile viewing the full-view spherical image data 200 displayed on thedisplay terminal 50, in which the full-view spherical image data 200transmitted from the robot 10 is required to be real time image. If thecommunication delay due to the congestion of the communication network 9occurs, the full-view spherical image data 200 is displayed on thedisplay terminal 50 with a time lag from the actual state around therobot 10, causing the degradation of the operability to the operator whoremotely controls the robot 10. Therefore, the remote-control system 1 acan prevent the deterioration of the operability, caused by the time lagof the display screen due to the communication delay, by changing theimage quality of the full-view spherical image data 200 transmitted fromthe robot 10 depending on the movement state of the robot 10.

The conditions included in the condition table 5002 are not limited tothe conditions related to the traveling velocity of the robot 10, butcan be any condition related to the movement state that the robot 10 canacquire. For example, the condition table 5002 can include conditionsrelated to the traveling direction of the robot 10.

Hereinafter, a description is given of an example of an operationcontrol of the robot 10 based on an input command input by an operatoroperating the robot 10 using the display terminal 50 with reference toFIG. 32. FIG. 32 is an example of a flowchart illustrating a robotcontrol process based on an input command at the display terminal 50according to the first embodiment. FIG. 32 illustrates a case in whichthe display screen 600 a (FIG. 27) is displayed on the display 512 ofthe display terminal 50. The display screen being displayed on thedisplay 512 of the display terminal 50 may be another display screenillustrated in FIG. 29 or FIG. 30.

In step S61 b, if the operation input reception unit 52 of the displayterminal 50 receives a specific operation input to the display screen600 a displayed on the display 512 (S61 b: YES), the operation inputreception unit 52 proceeds the sequence to step S62 b. For example, theoperation input reception unit 52 receives an operation input to any oneof the keys of the movement instruction key 605 included in the displayscreen 600 a. On the other hand, if the operation input reception unit52 does not receive the specific operation input to the display screen600 a (S61 b: N0), the sequence repeats step S61 b.

In step S62 b, the storing/reading unit 57 of the display terminal 50reads out the user command table 5003 (FIG. 20) stored in the storageunit 5000.

In step S63 b, the determination unit 54 of the display terminal 50searches the user command table 5003 read out by the storing/readingunit 57 and specifies an input command corresponding to the operationinput received by the operation input reception unit 52.. Then, thedetermination unit 54 searches the user command table 5003 to extractthe processing corresponding to the specified input command. Forexample, if an input to the “forward(↑)” key of the movement instructionkey 605 is received by the operation input reception unit 52, thedetermination unit 54 specifies the “pressing of “forward(↑)” key as theinput command. Then, the determination unit 54 extracts “MOVE (10.0,10.0)” as a process corresponding to the specified input command.

In step S64 b, if the type of processing corresponding to the extractedprocessing is “transmit command,” the display terminal 50 proceeds thesequence to step S65 b.

In step S65 b, the request command generation unit 55 of the displayterminal 50 generates a request command corresponding to the processingextracted by the determination unit 54. For example, if the extractedprocessing is “MOVE (10.0, 10.0),” the request command generation unit55 generates the request command including “MOVE (10.0, 10.0).”

In step S66 b, the transmission/reception unit 51 of the displayterminal 50 transmits the request command generated by the requestcommand generation unit 55 to the robot 10. Then, thetransmission/reception unit 31 of the robot 10 receives the requestcommand in response to the input operation by the operator, from thedisplay terminal 50. When the robot 10 receives the request command, therobot 10 executes a command processing (see FIG. 33) which will bedescribed later.

On the other hand, if the type of processing corresponding to theextracted processing is not “transmit command” in step S64 b (S64 b:NO), the display terminal 50 proceeds the sequence to step S67 b.

In step S67 b, the display terminal 50 performs the processing extractedin step S63 b. For example, if the extracted processing is “VIEW_H_ANGLE=0, VIEW_V_ANGLE=0” corresponding to the pressing of the “home” button601, the display terminal 50 uses the display control unit 53 to changethe first image capture direction of the full-view spherical imagedisplayed in the first display field 610.

In the above step S63 b, the determination unit 54 extracts the specificprocessing using the user command table 5003, but the display terminal50 can be configured to extract or execute the specific processing basedon an event assigned to the movement instruction key 605.

Hereinafter, with reference to FIG. 33, a description is given of acontrol process (e.g., movement control process) performed by the robot10 using the request command transmitted from the display terminal 50 byperforming the processing illustrated in FIG. 31 or FIG. 32. FIG. 33 isan example of a flowchart illustrating a control process of the robot 10based on the request command transmitted from the display terminal 50according to the first embodiment.

In step S71, if the transmission/reception unit 31 of the robot 10receives the request command transmitted from the display terminal 50(S71: YES), the transmission reception unit 31 proceeds the sequence tostep S72. On the other hand, if the transmission/reception unit 31 doesnot receive the request command (S71: NO), the sequence repeats stepS71.

In step S72, the storing/reading unit 41 of the robot 10 reads out thecommand table 3001 (FIG. 17A) stored in the storage unit 3000.

In step S73, the determination unit 34 of the robot 10 searches thecommand table 3001 read out by the storing/reading unit 41 to extract aspecific processing corresponding to the request command received by thetransmission/reception unit 31.

In step S74, the robot 10 performs the specific processing extracted instep S73.

For example, if the request command received by thetransmission/reception unit 31 is “MOVE (10.0, 10.0),” the determinationunit 34 searches the command table 3001 to extract the processingcorresponding to the command name of “MOVE.” In this case, the extractedprocessing is “10.0° rotation of the left wheel and 10.0° rotation ofthe right wheel.” Then, the determination unit 34 notifies an executionrequest of the extracted processing to the movement control unit 38.Then, based on the processing extracted by the determination unit 34,the movement control unit 38 rotates the left wheel of the movementmechanism 17 of the robot 10 for 10.0° and rotates the right wheel ofthe movement mechanism 17 of the robot 10 for 10.0°. The notificationdestination of the execution request becomes different depending on theprocessing extracted by the determination unit 34. For example, if therequest command received by the transmission/reception unit 31 is “ARM,”the notification destination of the execution request becomes the armoperation control unit 39, and if the request command is “SHOOT,” “FPS”or “RESOLUTION”, the notification destination of the execution requestbecomes the imaging instruction unit 36.

As above described, the robot 10 can perform the processing or operationbased on the request command transmitted from the display terminal 50 byperforming the processing illustrated in FIG. 31 or FIG. 32, and cantransmit the image (full-view spherical image data 200 or detailed imagedata 250) captured and acquired based on the request command to thedisplay terminal 50. With this configuration, the remote-control system1 a can remotely control the operation of the robot 10 using the displayterminal 50, and the information (e.g., full-view spherical image data200, detailed image data 250) acquired by the robot 10 using theremote-control operation can be displayed on the display terminal 50.

Transmission and Reception of Detailed Image:

Hereinafter, a description is given of a process of displaying thedetailed image acquired by the general imaging apparatus 23 included inthe robot 10 on the display terminal 50 with referenced to FIGS. 34 to36. FIG. 34 is an example of a sequence diagram illustrating a processof displaying the detailed image in the remote-control system 1 aaccording to the first embodiment. FIG. 34 illustrates a case in whichthe display screen 600 a (FIG. 27) is displayed on the display 512 ofthe display terminal 50.

The display screen displayed on the display 512 of the display terminal50 can be another display screen illustrated in FIG. 29 or FIG. 30.

In step S91, the operation input reception unit 52 of the displayterminal 50 receives an operation input to the “shoot” button 603included in the display screen 600 a. If the operation input receptionunit 52 receives the operation input to the “shoot” button 603, therequest command generation unit 55 of the display terminal 50 generatesa request command including a command of “ARM” and a command of “SHOOT.”As indicated in the command table 3001 (FIG. 17A), the processing of therobot 10 corresponding to the “SHOOT” command is “imaging by the generalimaging apparatus 23.” That is, the request command generation unit 55generates a control signal indicating an imaging request to the generalimaging apparatus 23. Since the specific processing from the receptionof the operation input to the generation of the request command is thesame as the processing illustrated in FIG. 32, the description thereofwill be omitted.

In steps S92-1 and S92-2, the transmission/reception unit 51 of thedisplay terminal 50 transmits the request command generated by therequest command generation unit 55 to the robot 10 via the managementserver 90. In this case, since the request command generated by therequest command generation unit 55 includes the “SHOOT” command, thetransmission/reception unit 51 transmits (outputs) the control signalindicating the imaging request to the general imaging apparatus 23.Then, the transmission/reception unit 31 of the robot 10 receives therequest command from the display terminal 50 via the management server90.

In step S93, the arm operation control unit 39 of the robot 10transforms the manipulation arm 11 based on the request command receivedby the transmission/reception unit 31. The arm operation control unit 39changes the direction or orientation of the manipulation arm 11 bytransforming the manipulation arm 11 based on the “ARM” command includedin the request command received by the transmission/reception unit 31.

Hereinafter, the control operation of the manipulation arm 11 by the armoperation control unit 39 is described. The variable (h,v) of “ARM”command included in the request command indicates the relative positionon the screen in the first image capture direction of the full-viewspherical image data 200 displayed on the display screen 600 a. The “h”is the horizontal angle with respect to the traveling direction of therobot 10 and the “v” is the vertical angle in the vertical directionwith respect to the traveling direction of the robot 10. The first imagecapture direction of the full-view spherical image data 200 displayed onthe display screen 600 a is set to (h₀,v₀).

The arm operation control unit 39 corrects the variable (h,v) based onthe first image capture direction (h₀,v₀) of the full-view sphericalimage data 200 to change the direction or orientation of themanipulation arm 11 based on the corrected variable. For example, whenthe center in the first image capture direction of the full-viewspherical image data 200 displayed on the display screen 600 a isspecified by the variable (h,v) of the “ARM” command, (h,v)=(h₀,v₀) isset. On the other hand, if any position other than the center in thefirst image capture direction of the full-view spherical image data 200displayed on the display screen 600 a is specified by the variable (h,v)of the “ARM” command, (h, v) is corrected using the following [Math. 2]in which the deviation from the center is set as “α” and “β” by setting“α>0” and “β>0” for the upper right direction of the display screen 600a. In [Math. 2]“a” and “b” are specific coefficients.

h=h ₀ +α×a

v=v ₀ +β×b   [Math. 2]

In this configuration, by specifying the variable indicating therelative position on the screen in the first image capture direction ofthe full-view spherical image data 200 displayed on the display screen600 a using the “ARM” command included in the request command, the annoperation control unit 39 can change the direction or orientation of themanipulation arm 11 by transforming the manipulation arm 11 based on thespecified variable (h,v). That is, the second image capture direction tobe used by the general imaging apparatus 23 is determined based on thefirst image capture direction of the full-view spherical image data 250,and the robot 10 changes the direction or orientation of themanipulation arm 11 into a direction or orientation that the generalimaging apparatus 23 can capture images in the second image capturedirection determined by the arm operation control unit 39. For example,when the upper right of the display screen is designated by the variable(h,v) of the “ARM” command, the arm operation control unit 39 can changethe direction or orientation of the manipulation arm 11 to the uppertight direction for the specified variable. In a case that the verticaldirection of the manipulation arm 11 cannot be changed due to the shapeof the manipulation arm 11, the arm operation control unit 39 can changethe horizontal direction of the manipulation arm 11 alone using only “h”of the variable (h,v) of the “ARM” command included in the requestcommand.

In step S94, the imaging instruction unit 36 of the robot 10 outputs animaging instruction to the general imaging apparatus 23 in response to acompletion of changing the direction or orientation of the manipulationarm 11 by the arm operation control unit 39. Then, the general imagingapparatus 23 performs the imaging process in response to the imaginginstruction output from the imaging instruction unit 36. In this case,the general imaging apparatus 23 acquires the detailed image data 250,which is a detailed image captured in the direction or orientation ofthe manipulation arm 11 changed by the arm operation control unit 39.

In step S95, the image acquisition unit 37 of the robot 10 acquires thedetailed image data 250 acquired by the general imaging apparatus 23,from the general imaging apparatus 23.

In step S96, the storing/reading unit 41 of the robot 10 stores thedetailed image data 250 acquired by the image acquisition unit 37 in thestorage unit 5000.

In steps S97-1 and S97-2, the transmission/reception unit 31 of therobot 10 transmits the detailed image data 250 acquired by the imageacquisition unit 37 to the display terminal 50 via the communicationnetwork 9. Then, the transmission/reception unit 51 of the displayterminal 50 receives the detailed image data 250 transmitted from therobot 10. It should be noted that the order of processing of step S96,and steps S97-1 and S97-2 can be changed or can be performed in parallelwith each other.

In step S98, the storing/reading unit 57 of the display terminal 50stores the detailed image data 250 received by thetransmission/reception unit 51 in the storage unit 5000.

In step S99, the display control unit 53 of the display terminal 50displays, on the display 512,the detailed image data 250 received by thetransmission/reception unit 51. It should be noted that the order ofprocessing of step S98 and step S99 can be changed or can be performedin parallel with each other.

FIG. 35 illustrates an example of a display screen for displaying thedetailed image data 250 transmitted from the robot 10. As to the displayscreen 650 illustrated in FIG. 35, the detailed image data 250transmitted from the robot 10 is displayed in the second display field630. The display screen 650 also includes, for example, a “previousimage” button 651 and a “next image” button 652.

If the operation input reception unit 52 receives an input operation tothe “previous image” button 651, the display control unit 53 displaysthe detailed image data 250 corresponding to a previous image (previousdetailed image data) that was received just before receiving thedetailed image data 250 corresponding to an image (detailed image datareceived just after previous detailed image data) being currentlydisplayed in the second display field 630, in which these detailed imagedata are stored in the storage unit 5000.

Further, if the operation input reception unit 52 receives an inputoperation to the “next image” button 652, the display control unit 53displays the detailed image data 250 corresponding to a next image (nextdetailed image data) that was received just after receiving the detailedimage data 250 corresponding to an image (detailed image data receivedjust before next detailed image data) being currently displayed in thesecond display field 630, in which these detailed image data are storedin the storage unit 5000.

The operator who operates the robot 10 using the display terminal 50 canselect the respective tabs of “full-view spherical image” and “detailedimage” set at the upper left of the first display field 610 and thesecond display field 630 to change the image display between of thefull-view spherical image and the detailed image. For example, when thedisplay screen 650 is displaying the detailed image on the displayterminal 50, the operator can switch the display screen 650 to thedisplay screen of the full-view spherical image data 200 (e.g., displayscreen 600 a of FIG. 27) by selecting the tab of “full-view sphericalimage” set at the upper left of the first display field 610.

Although the configuration of switching the image display between thefull-view spherical image data 200 and the detailed image data 250 basedon the selection of the tabs is described, the display screen 650 can beconfigured to simultaneously display the full-view spherical image data200 and the detailed image data 250. Further, the display terminal 50can be configured to display the full-view spherical image data 200 asone screen and the detailed image data 250 as another screen, anddisplay one screen over another screen or another screen over one screenby overlapping one screen and another screen at least partially.

In the above described configuration, the display of the detailed imagedata 250 stored in the storage unit 5000 is switched using the selectionof the “previous image” button 651 and the “next image” button 652, butnot limited thereto. For example, the display screen 650 can beconfigured to display an image list of the detailed image data 250stored in the storage unit 5000 to allow the user to select the detailedimage data 250 to be displayed.

In the above described configuration, the detailed image data 250captured by the general imaging apparatus 23 can be displayed on thedisplay terminal 50 when the operator presses the “shoot” button 603,but not limited thereto. For example, the line-of-sight position of theoperator who operates the robot 10 using the display terminal 50 can bedetected, and then the image capture position can be specified and thenthe imaging instruction can be performed.

In a display screen 600 e illustrated in FIG. 36, a line-of-sight icon619 indicating a line-of-sight position (viewing position) of theoperator is displayed in the first display field 610 instead of the“shoot” button 603. The line-of-sight detection unit 56 of the displayterminal 50 detects the line-of-sight position (e.g., viewpoint) of theoperator using the line-of-sight detection device 517. Then, the displaycontrol unit 53 displays the line-of-sight icon 619 at the line-of-sightposition detected by the line-of-sight detection unit 56 bysuperimposing the line-of-sight icon 619 over the full-view sphericalimage data 200 being displayed on the first display field 610.

In this case, if the line-of-sight position detected by theline-of-sight detection unit 56 does not move for a given time period,the request command generation unit 55 generates a request commandincluding the “ARM” command designating the detected line-of-sightposition as the variable (h,v) and the “SHOOT” command, which is theimaging instruction transmitted to the general imaging apparatus 23. Thevariable (h,v) of the “ARM” command indicates the relative positions ofthe detected line-of-sight position on the screen in the first imagecapture direction of the full-view spherical image data 200 displayed onthe display screen 600 a. Then, the transmission/reception unit 51 ofthe display terminal 50 transmits the generated request command to therobot 10. By performing the same processing in step S93 and subsequentsteps in FIG. 34, the display terminal 50 can display the detailed imagedata 250 at the line-of-sight position detected by the line-of-sightdetection unit 56.

In the above described configuration, the image capture position isspecified (identified) and the imaging instruction is performed based onthe line-of-sight position detected by the line-of-sight detection unit56, but not limited thereto. For example, the display terminal 50 can beconfigured to specify the image capturing position based on theline-of-sight position detected by the line-of-sight detection unit 56and to transmit the imaging instruction by performing the inputoperation to the “shoot” button 603. Further, the display terminal 50can be configured to specify the image capturing position using aposition of the mouse pointer 620 and to transmit the imaginginstruction when the line-of-sight position detected by theline-of-sight detection unit 56 does not move for a given time period.

In the above described remote-control system 1 a, if the operator whoremotely controls the robot 10 by viewing the full-view spherical imagedata 200 displayed on the display terminal 50 wants to check the detailsof a specific region in the site where the robot 10 is located, thedetailed image data 250, which is acquired by capturing a part orportion of an object existing in the full-view spherical image data 200,can be displayed on the display terminal 50. Therefore, as to the abovedescribed remote-control system 1 a, the operator who operates the robot10 using the display terminal 50 can check the detail information of thespecific region.

Further, as described above, since the detailed image is the imageacquired by the general imaging apparatus 23 using the lens having afocal length longer than a focal length of the lens used for the specialimaging apparatus 21, the detailed image becomes the image having higherimage quality compared to the image quality of the full-view sphericalimage acquired by the special imaging apparatus 21. Therefore, as to theabove described remote-control system 1 a, the operator who operates therobot 10 using the display terminal 50 can check the detail informationof the specific region using a higher quality image, such as higherresolution image.

The above described remote-control system 1 a can be effectively used,for example, when a maintenance work of devices or the like disposed ata remote site is performed using the robot 10. For example, the operatorof the display terminal 50 performs a movement operation of the robot 10by viewing the full-view spherical image data 200 transmitted from therobot 10 and requests the imaging of a concerned portion to the generalimaging apparatus 23 if the concerned portion is found. With thisconfiguration, based on the operator needs, the remote-control system 1a can switch the image display between the full-view spherical imagedata 200, capable of viewing a wider range of the circumference orsurroundings of the robot 10, and the detailed image data 250 capable ofviewing the circumference or surroundings of the robot 10 with thehigher image quality, which can be displayed on the display terminal 50.Therefore, the remote-control system 11.a can achieve both animprovement in the operability of the robot 10 and an improvement in theresolution of the image to be displayed on the display terminal 50 whena specific region is to be checked in detail.

As to the above described configuration, the captured image of thespecific region can be displayed on the display terminal 50 by cuttingout the specific region in the full-view spherical image data 200 orenlarging the specific region in the full-view spherical image data 200using the zooming function. However, if the image is displayed on thedisplay terminal 50 using this method using the zooming function, thezoomed image has a lower resolution even compared to the full-viewspherical image, and thereby the operator cannot confirm the details inthe full-view spherical image. Therefore, in the remote-control system 1a, the image acquired by the imaging apparatuses (special imagingapparatus 21 or the general imaging apparatus 23) having the differentimaging purposes can be switched when displaying on the display terminal50, with which both of the improvement of operability and theimprovement of the resolution (image resolution) can be achieved.

Switching of Display Screen

Hereinafter, a description is given of a process of switching the imagedisplay between the full-view spherical image and the detailed image atthe display terminal 50 based on the movement state of the robot 10.FIG. 37 is an example of a sequence diagram illustrating a process ofswitching an image displayed on the display terminal 50 in anenvironment of the remote-control system 1 a according to the firstembodiment. FIG. 37 illustrates a case when the display screen 600 a(displaying full-view spherical image data 200) illustrated in FIG. 27is currently displayed on the display 512 of the display terminal 50.

In steps S101 a-1 and S101 a-2, the transmission/reception unit 31 ofthe robot 10 transmits the state information 150 (FIG. 28) generated bythe state information generation unit 35 to the display terminal 50using a communication session established with the management server 90.Then, the transmission/reception unit 51 of the display terminal 50receives (acquires) the state information 150 from the robot 10 via themanagement server 90 (an example of acquisition step).

In step S102 a, if the robot 10 is stopped (S102 a: YES), the displayterminal 50 proceeds the sequence to step S103 a. Specifically, thedetermination unit 54 of the display terminal 50 determines that therobot 10 is stopped when a value of the variable of “SPEED” included inthe state information 150 received by the transmission/reception unit 51is “0 km/h.” It should be noted that the numerical value of the variableof “SPEED” used for determining that the robot 10 is stopped is notlimited to “0 km/h.” For example, the determination unit 54 of thedisplay terminal 50 can be configured to determine that the robot 10 isstopped if the numerical value of the variable of “SPEED” becomes lessthan or equal to a specific threshold value.

On the other hand, in step S102 a, if the determination unit 54 of thedisplay terminal 50 determines or detects that the robot 10 is notstopped but is moving (S102 a: NO), the determination unit 54 terminatesor ends the sequence.

In step S103 a, the request command generation unit 55 of the displayterminal 50 generates a request command including the “ARM” command andthe “SHOOT” command. As illustrated in the command table 3001 (FIG.17A), the processing of the robot 10 corresponding to the “SHOOT”command is “imaging by the general imaging apparatus 23.” That is, therequest command generation unit 55 generates a control signal indicatingan imaging request to the general imaging apparatus 23. Further, thevariable (h,v) of the “ARM” command included in the request commandcorresponds to the center position on the screen in the first imagecapture direction of the full-view spherical image data 200 displayed onthe display screen 600 a. Further, the variable (h,v) of the “ARM”command may correspond to a position specified by the mouse pointer 620on the screen in the first image capture direction of full-viewspherical image data 200 displayed on the display screen 600 a.

In steps S104 a-1 and S104 a-2, the transmission/reception unit 51 ofthe display terminal 50 transmits the request command (“ARM” and“SHOOT”) generated by the request command generation unit 55 to therobot 10 via the management server 90. In this case, since the requestcommand generated by the request command generation unit 55 includes the“SHOOT” command, the transmission/reception unit 51 transmits (outputs)a control signal indicating an imaging request to the general imagingapparatus 23. Then, the transmission/reception unit 31 of the robot 10receives the request command (“ARM” and “SHOOT”) from the displayterminal 50 via the management server 90.

In step S105 a, the arm operation control unit 39 of the robot 10transforms the manipulation arm 11 based on the request command receivedby the transmission/reception unit 31. Then, the arm operation controlunit 39 changes the direction or orientation of the manipulation arm 11by transforming the manipulation arm 11 based on the “ARM” commandincluded in the request command received by the transmission/receptionunit 31, in which a position of the manipulation arm 11 may be changed.Since the process of changing the direction or orientation of themanipulation arm 11 by the arm operation control unit 39 is the same asthe processing in step S93 of FIG. 34, the description thereof will beomitted.

In step S106 a, the imaging instruction unit 36 of the robot 10 outputsan instruction information indicating an imaging instruction to thegeneral imaging apparatus 23 in response to a completion of changing thedirection or orientation of the manipulation arm 11 by the arm operationcontrol unit 39. Then, the general imaging apparatus 23 performs theimaging process in response to the imaging instruction output from theimaging instruction unit 36. In this case, the general imaging apparatus23 acquires the detailed image data 250, which is the detailed imagecaptured in the direction or orientation of the manipulation arm 11changed by the arm operation control unit 39. In step S107 a, the imageacquisition unit 37 of the robot 10 acquires the detailed image data 250captured by the general imaging apparatus 23, from the general imagingapparatus 23.

In step S108 a, the storing/reading unit 41 of the robot 10 stores thedetailed image data 250 acquired by the image acquisition unit 37 in thestorage unit 5000.

In step S109 a-1 and S109 a-2, the transmission/reception unit 31 of therobot 10 transmits the detailed image data 250 acquired by the imageacquisition unit 37 to the display terminal 50 via the management server90. Then, the transmission/reception unit 51 of the display terminal 50receives the detailed image data 250 transmitted from the robot 10. Itshould be noted that the processing order of steps 108 a and S109 a-1and S109 a-2 can be changed or can be performed in parallel with eachother.

In step S110 a, the storing/reading unit 57 of the display terminal 50stores the detailed image data 250 acquired by thetransmission/reception unit 51 in the storage unit 5000.

In step S111 a, the display control unit 53 of the display terminal 50displays the detailed image data 250, received by thetransmission/reception unit 51, on the display 512. With thisconfiguration, the display control unit 53 switches the image displayedon the display 512 from the full-view spherical image data 200 (e.g.,display screen 600 a in FIG. 27) to the detailed image data 250 (e.g.,display screen 650 in FIG. 35). That is, the display terminal 50 outputsthe detailed image data 250 based on the state information 150 received(acquired) by the processing in step S101 a-2. It should be noted thatthe order of processing in step S110 a and step S111 a can be changed orcan be performed in parallel with each other.

As above described, the remote-control system 1 a can switch the displayof the full-view spherical image data 200 and the detailed image data250 displayed on the display terminal 50 based on the movement state ofthe robot 10. For example, when the robot 10 is moving, the displayterminal 50 displays the full-view spherical image data 200, capable ofviewing a wider range of the circumference or surroundings of the robot10, and when the robot 10 is stopped, the display terminal 50 displaysthe detailed image data 250, capable of viewing the circumference orsurroundings of the robot 10 with the higher image quality. Therefore,for example, when the robot 10 is moving, the remote-control system 1 acan assist the operator to operate the robot 10 by checking the ground(foot area) and the surroundings of the robot 10, and can assist theoperator to move the robot 10 without colliding with surrounding personsand objects, with which the operability of the robot 10 can be improved.

Further, since the probability that the robot 10 collides withsurrounding persons and objects is lower when the robot 10 is stopped,the remote-control system la can assist the operator operating the robot10 to confirm a specific region around the robot 10 with higher imagequality by displaying the detailed image data 250 on the displayterminal 50. Therefore, the remote-control system 1 a can improve theoperability of the robot 10 and the resolution of the displayed image byswitching the image display on the operation screen (display screen)based on the movement state of the robot 10.

Further, the display terminal 50 can be configured to generate therequest command if the determination unit 54 has determined that therobot 10 is stopped in step S102 a and then the determination unit 54determines or detects that the robot 10 is not stopped (i.e., the robot10 is moving again) after the determination unit 54 has determined thatthe robot 10 is stopped in step S102 a.

Further, the display terminal 50 can be configured to generate therequest command if the determination unit 54 has determined that therobot 10 is stopped in step S102 a and then a given time period elapsesafter the determination unit 54 has determined that the robot 10 isstopped in step S102 a.

With this configuration, even if the robot 10 is being stopped at thesame position, the remote-control system 1 a can reduce the probabilitythat the detailed image data 250 is acquired by the general imagingapparatus 23 and then displayed on the display terminal 50, with whichwasteful processing such as acquiring the detailed image data 250 whilethe robot 10 is being stopped at the same position can be reduced.Further, with this configuration, the remote-control system 1 a canreduce the processing load of the robot 10 and the display terminal 50and prevent the occurrence of communication delay caused by thecongestion of the communication network 9.

Example of Another Display Screen:

Hereinafter, a description is given of another example of the displayscreen displayed on the display terminal 50. The display terminal 50 canbe configured to display different display screens depending on thetypes of display 512 provided for the display terminal 50. FIG. 38 is anexample of a display screen 670 displayed on a head-mount display (HMI))used as an example of the display terminal 50. The display screen 670illustrated in FIG. 38 is an example of a screen displayed on the headmount display (HMD), in which the full-view spherical image data 200 andthe detailed image data 250 transmitted from the robot 10 are displayed.

In the display screen 670 illustrated in FIG. 38, a display field of thefull-view spherical image data 200 is set in the entire display screen670, and the second display field 630 of the detailed image data 250 aresuperimposed at least partially on the display screen 670. An operatorfitted with the head mount display (HMD) displaying the display screen670 can view any direction in the full-view spherical image data 200 bychanging the head orientation of the operator. Further, since variousbuttons or icons displayed on the display screen 670 move in accordancewith the head orientation of the operator, even if the first imagecapture direction of the full-view spherical image data 200 is changed,various buttons or icons can be displayed at specific positions on thedisplay screen 670.

As to the above described first embodiment, the display terminal 50 canbe used to control the robot 10 (an example of movable apparatus)including the special imaging apparatus 21 (an example of first imagingapparatus), which captures images of objects and acquiring the full-viewspherical image data 200 (an example of first image), and the generalimaging apparatus 23 (an example of second imaging apparatus), whichcaptures a part or portion of the objects captured by the specialimaging apparatus 21 (an example of second image). Based on the stateinformation 150 indicating the movement state of the robot 10, thedisplay terminal 50 switches the image displayed on the display 512 (anexample of the display unit) between the full-view spherical image data200 and the detailed image data 250. Therefore, the display terminal 50can implement advanced control when the remote control is performed forthe robot 10 equipped with the special imaging apparatus 21 and thegeneral imaging apparatus 23.

Further, when the robot 10 is stopped, the display terminal 50 of thefirst embodiment displays the detailed image data 250 (an example ofsecond image) on the display 512 (an example of display unit).Therefore, for example, when the robot 10 is moving, the displayterminal 50 displays the full-view spherical image data 200, with whichthe robot 10 can be operated while preventing the robot 10 fromcolliding with surrounding persons and objects by confirming the ground(foot area) and the surroundings of the robot 10, so that theoperability of the robot 10 can be improved. Further, since theprobability that the robot 10 collides with surrounding persons andtargets (e.g., objects) is lower when the robot 10 is stopped, theremote-control system la can assist the operator operating the robot 10to confirm a specific region around the robot 10 with higher imagequality by displaying the detailed image data 250 on the displayterminal 50. Therefore, the display terminal 50 can improve theoperability of the robot 10 and the resolution of the display image byswitching the display of the operation screen (display screen) based onthe movement state of the robot 10.

Further, the display terminal 50 of the first embodiment displays thedetailed image data 250 (an example of second image), acquired byimaging the second image capture direction determined based on the firstimage capture direction of the full-view spherical image data 200 (anexample of first image) displayed on the display 512 (an example ofdisplay unit), using the general imaging apparatus 23 (an example ofsecond imaging apparatus). Therefore, the display terminal 50 can allowthe operator who operates the robot 10 using the display terminal 50 toconfirm the details of a part or portion of the targets (e.g., objects)included in the full-view spherical image data 200 displayed on thedisplay 512.

Further, when the robot 10 is moving, the display terminal 50 of thefirst embodiment displays the full-view spherical image data 200 (anexample of first image) having reduced image quality on the display 512(an example of display unit). With this configuration, the displayterminal 50 can prevent the deterioration of the operability caused bythe time lag of the display screen due to the communication delay andthe like.

Further, the display terminal 50 of the first embodiment displays thevideo image data on the display 512 (an example of display unit) as thefull-view spherical image data 200 (an example of first image) anddisplays the still image data on the display 512 as the detailed imagedata 250 (an example of second image). Therefore, the display terminal50 can allow the operator who operates the robot 10 using the displayterminal 50 to perform the remote control of the robot 10 by checkingthe video image data of the full-view spherical image that is played onthe display terminal 50 using the streaming play, and can display thedetailed information of a specific region using the still image data ofthe detailed image, with which the operability of the robot 10 can beimproved.

In the remote-control system 1 a according to the first embodiment, thedisplay terminal 50 is an example of an output control apparatus (orcontrol apparatus) according to the embodiment of the present invention.The display terminal 50 used as the output control apparatus controlsthe robot 10 (an example of movable apparatus) including the specialimaging apparatus 21 (an example of first imaging apparatus), whichcaptures images of objects and acquires the full-view spherical imagedata 200 (an example of first image), and the general imaging apparatus23 an example of the second imaging apparatus), which captures a part orportion of the objects captured by the special imaging apparatus 21 (anexample of second image), and outputs the full-view spherical image data200 and the detailed image data 250. Then, the display terminal 50 usedas the output control apparatus acquires (receives) the stateinformation 150 indicating the movement state of the robot 10, andoutputs (displays) the full-view spherical image data 200 and thedetailed image data 250 selectively based on the acquired stateinformation 150. Therefore, the display terminal 50 used as the outputcontrol apparatus can implement advanced control when the remote controlis performed for the robot 10 equipped with the special imagingapparatus 21 and the general imaging apparatus 23.

Further, in the remote-control system 1 a according to the firstembodiment, the display terminal 50 is an example of imaging controlapparatus according to the first embodiment. The display terminal 50used as the imaging control apparatus controls the robot 10 includingthe special imaging apparatus 21 (an example of first imagingapparatus), which captures images of objects and acquires the full-viewspherical image data 200 (an example of first image), and the generalimaging apparatus 23 (an example of second imaging apparatus), whichcaptures a part or portion of the objects captured by the specialimaging apparatus 21 (an example of the second image). Then, the displayterminal 50 used as the imaging control apparatus receives (acquires)the state information 150 indicating the movement state of the robot 10,and outputs (transmits) a request command (an example of control signal)indicating the imaging request to the general imaging apparatus 23 basedon the acquired state information 150. Therefore, the display terminal50 used as the imaging control apparatus can implement advanced controlwhen the remote control is performed for the robot 10 equipped with thespecial imaging apparatus 21 and the general imaging apparatus 23.

Second Embodiment

Hereinafter, a description is given of a remote-control system 1 baccording to a second embodiment. The same configuration and the samefunction as those of the first embodiment are denoted by the samereference numerals, and the description thereof will be omitted. Theremote-control system 1 b according to the second embodiment is a systemthat the control device 30 included in the robot 10 (i.e., robot 10itself) determines the image (i.e., full-view spherical image ordetailed image) to be transmitted (output) to the display terminal 50based on the movement state of the robot 10.

The hardware configuration and the functional configuration of theapparatus or terminal (e.g., robot 10, display terminal 50, managementserver 90) configuring the remote-control system 1 b according to thesecond embodiment are the same as those of the apparatus or terminal(e.g., robot 10, display terminal 50, management server 90) configuringthe remote-control system 1 a according to the first embodiment, andthereby the description thereof will be omitted.

FIG. 39 is an example of a sequence diagram illustrating a process ofswitching the image displayed on the display terminal 50 in anenvironment of the remote-control system 1 b according to the secondembodiment. In FIG. 39, the processing performed by the control device30 included in the robot 10 will be described as the processingperformed by the robot 10. Further, FIG. 39 illustrates an example casethat the display screen 600 a (displaying full-view spherical image data200) illustrated in FIG. 27 is displayed on the display 512 of thedisplay terminal 50.

In step S101 b, the state information generation unit 35 of the robot 10generates the state information 150 (FIG. 28) indicating the movementstate of the robot 10 based on the drive state of the movement mechanism17 acquired from the movement control unit 38. With this configuration,the state information generation unit 35 acquires the state information150 (an example of acquisition step).

In step S102 b, if the determination unit 34 of the robot 10 determinesthat the robot 10 is stopped based on the state information 150 (S102bYES), the determination unit 34 proceeds the sequence to step S103 b.Specifically, the determination unit 34 of the robot 10 determines ordetects that the robot 10 is stopped when the numerical value of thevariable of “SPEED” included in the state information 150 generated(acquired) by the state information generation unit 35 is “0 km/h.” Itshould be noted that the value of the variable of “SPEED” used fordetermining that the robot 10 is stopped is not limited to “0 km/h.” Forexample, the determination unit 34 of the robot 10 can be configured todetermine or detect that the robot 10 is stopped if the numerical valueof the variable of “SPEED” is less than or equal to a specific thresholdvalue.

On the other hand, in step S102 b, if the determination unit 34 of therobot 10 determines or detects that the robot 10 is not stopped but ismoving based on the state information 150 (S102 bNO), the determinationunit 34 terminates or ends the sequence.

In step S103 b, the arm operation control unit 39 of the robot 10transforms the manipulation arm 11 based on the state information 150generated (acquired) by the state information generation unit 35.Specifically, the arm operation control unit 39 changes the direction ororientation of the manipulation arm 11 in the direction indicated by thevalues of the variables of “H_ANGLE” and “V_ANGLE” included in the stateinformation 150. That is, the arm operation control unit 39 changes thedirection or orientation of the manipulation arm 11 in the samedirection of the traveling direction of the robot 10.

In step S104 b, the imaging instruction unit 36 of the robot 10 outputsan instruction information indicating the imaging instruction to thegeneral imaging apparatus 23 in response to a completion of changing thedirection or orientation of the manipulation arm 11 by the arm operationcontrol unit 39. Then, the general imaging apparatus 23 performs theimaging process in response to the imaging instruction output from theimaging instruction unit 36. In this case, the general imaging apparatus23 acquires the detailed image data 250, which is a detailed imagecaptured in the direction or orientation of the manipulation arm 11changed by the arm operation control unit 39.

In step S105 b, the image acquisition unit 37 of the robot 10 acquiresthe detailed image data 250 acquired by the general imaging apparatus23, from the general imaging apparatus 23.

In step S106 b, the storing/reading unit 41 of the robot 10 stores thedetailed image data 250 acquired by the image acquisition unit 37 in thestorage unit 3000.

In steps S106 b-1 and S106 b-2, the transmission/reception unit 31 ofthe robot 10 transmits the detailed image data 250 acquired by the imageacquisition unit 37 to the display terminal 50 via the management server90. That is, the robot 10 outputs the detailed image data 250 based onthe state information 150 generated (acquired) by the processing in stepS101 b. Then, the transmission/reception unit 51 of the display terminal50 receives the detailed image data 250 transmitted from the robot 10via the management server 90. It should be noted that the order ofprocessing in step S106 b and steps S106 b-1 and S106 b-2 can be changedor can be performed in parallel with each other.

In step S107 b, the storing/reading unit 57 of the display terminal 50stores the detailed image data 250 received by thetransmission/reception unit 51 in the storage unit 5000.

In step S108 b, the display control unit 53 of the display terminal 50displays, on the display 512, the detailed image data 250 received bythe transmission/reception unit 51. It should be noted that the order ofsteps S107 b and S108 b can be changed or can be performed in parallelwith each other.

Further, the robot 10 can be configured to perform the transformationprocessing of the manipulation arm 11 if the determination unit 34 ofthe robot 10 has determined that the robot 10 is stopped in step S102 band then the determination unit 34 determines or detects that the robot10 is not stopped (i.e., the robot 10 is moving again) after thedetermination unit 34 has determined that the robot 10 is stopped instep S102 b.

Further, the robot 10 can be configured to perform the transformationprocessing of the manipulation arm 11 if the determination unit 34 ofthe robot 10 has determined that the robot 10 is stopped in step S102 band then a given time period elapses after the determination unit 34 hasdetermined that the robot 10 is stopped in step S102 b.

With this configuration, even if the robot 10 is being stopped at thesame position, the remote-control system 1 b can reduce the probabilitythat the detailed image data 250 is acquired by the general imagingapparatus 23 and then displayed on the display terminal 50, with whichwasteful processing such as acquiring the detailed image data 250 whilethe robot 10 is being stopped at the same position can be reduced.Further, with this configuration, the remote-control system 1 b canreduce the processing load of the robot 10 and the display terminal 50and prevent the occurrence of communication delay caused by thecongestion of the communication network 9.

As to the above described remote-control system 1 b of the secondembodiment, the control device 30 (an example of information processingapparatus) included in the robot 10 (an example of movable apparatus)determines the image (i.e., the full-view spherical image data 200 ordetailed image data 250) to be transmitted to the display terminal 50based on the movement state of the robot 10. Therefore, as to theremote-control system 1 b, by transmitting an appropriate image from therobot 10 to the display terminal 50 based on the movement state of therobot 10, the occurrence of time lag of the display screen due tocommunication delay can be prevented, and by displaying an imagesuitable for the operator of the robot 10 on the display terminal 50,the remote control of the robot 10 including the special imagingapparatus 21 and the general imaging apparatus 23 can be implementedwith higher accuracy.

Further, the control device 30 (an example of information processingapparatus) included in the robot 10 according to the second embodimentcan communicate with, via the communication network 9, the displayterminal 50 that controls the robot 10 (an example of movable apparatus)including the special imaging apparatus 21 (an example of first imagingapparatus), which captures images of objects and acquires the full-viewspherical image data 200 (an example of first image), and the generalimaging apparatus 23 (an example of second imaging apparatus), whichcaptures a part or portion of the objects captured by the specialimaging apparatus 21 (an example of second image). Then, based on themovement state of robot 10, the control device 30 transmits thefull-view spherical image data 200 or the detailed image data 250,acquired from the special imaging apparatus 21 or the general imagingapparatus 23, to the display terminal 50 to switch the image displaybetween the full-view spherical image data 200 and the detailed imagedata 250 displayed on the display terminal 50. Therefore, the controldevice 30 can implement advanced control when the remote control isperformed for the robot 10 equipped with the special imaging apparatus21 and the general imaging apparatus 23.

In the remote-control system 1 b according to the second embodiment, thecontrol device 30 included in the robot 10 is an example of an outputcontrol apparatus according to the embodiment of the present invention.The control device 30 (an example of information processing apparatus)used as the output control apparatus controls the robot 10 (an exampleof movable apparatus) including the special imaging apparatus 21 (anexample of first imaging apparatus), which captures images of objectsand acquires the full-view spherical image data 200 (an example of firstimage), and the general imaging apparatus 23 (an example of secondimaging apparatus), which captures a part or portion of the objectscaptured by the special imaging apparatus 21 (an example of secondimage). Then, the control device 30 used as the output control apparatusacquires the state information 150 indicating the movement state of therobot 10, and outputs (transmits) the full-view spherical image data 200and the detailed image data 250 selectively based on the acquired stateinformation 150. Therefore, the control device 30 used as the outputcontrol apparatus can implement advanced control when the remote controlis performed for the robot 10 equipped with the special imagingapparatus 21 and the general imaging apparatus 23.

In the remote-control system 1 b according to the second embodiment, thecontrol device 30 included in the robot 10 is an example of the imagingcontrol apparatus according to the second embodiment. The control device30 (an example of information processing apparatus) used as the imagingcontrol apparatus controls the robot 10 (an example of movableapparatus) including the special imaging apparatus 21 (an example offirst imaging apparatus), which captures images of objects and acquiresthe fill-view spherical image data 200 (an example of first image), andthe general imaging apparatus 23 (an example of second imagingapparatus), which captures a part or portion of the objects captured bythe special imaging apparatus 21 (an example of second image). Then, thecontrol device 30 used as the imaging control apparatus acquires thestate information 150 indicating the movement state of the robot 10, andoutputs an instruction information (an example of control signalindicating an imaging request) for instructing the imaging request tothe general imaging apparatus 23 based on the acquired state information150. Therefore, the control device 30 used as the imaging controlapparatus can implement advanced control when the remote control isperformed using the robot 10 equipped with the special imaging apparatus21 and the general imaging apparatus 23.

Third Embodiment

Hereinafter, a description is given of a remote-control system 1 caccording to a third embodiment. The same configuration and the samefunction as those of the first or second embodiments are denoted by thesame reference numerals, and the description thereof will be omitted.The remote-control system 1 c according to the third embodiment is asystem using an image processing server 70 to determine the image (i.e.,the full-view spherical image or detailed image) to be transmitted tothe display terminal 50 based on the movement state of the robot 10.

System Configuration:

At first, with reference to FIG. 40, a description is given of aconfiguration of the remote-control system 1 c according to the thirdembodiment. FIG. 40 illustrates an example of a system configuration ofthe remote-control system 1 c according to the third embodiment. Asillustrated in FIG. 40, the remote-control system 1 c of the thirdembodiment is devised by adding the image processing server 70 to theconfiguration illustrated in FIG. 1. The image processing server 70 iscommunicably connected to the robot 10, the display terminal 50 and themanagement server 90 via the communication network 9. The imageprocessing server 70 transmits and receives image data with the robot 10and the display terminal 50 using a communication session established bythe management server 90.

The image processing server 70 may be a single server, or the imageprocessing server 70 can be configured with a plurality of servers toperform the distributed image processing. The image processing server 70performs the image processing on the fill-view spherical image data 200transmitted from the robot 10 based on the movement state of the robot10, and transmits the processed image data to the display terminal 50.The image processing server 70 can be configured by a plurality ofservers, and the function can be provided to any server. The imageprocessing server 70 is an example of information processing apparatusin this description.

The image processing server 70 and the management server 90 can beconfigured as a server system 7. The image processing server 70 and themanagement server 90 can be configured by a single server. Further, thedisplay terminal 50 and the image processing server 70 can be configuredas a display system 5. Further, the robot 10 (10A, 10B, 10C) and theimage processing server 70 can be configured as a movable apparatuscontrol system 3 (transmission control system)

Since the hardware configuration of the image processing server 70 isthe same as the hardware configuration of the management server 90illustrated in FIG. 15, the description thereof will not be omitted.Hereinafter, a description is given that the image processing server 70employs the hardware configuration illustrated in FIG. 15.

Functional Configuration:

FIGS. 41A and 41B (FIG. 41) illustrate an example of a functional blockdiagram of the remote-control system 1 c according to the thirdembodiment. The functions implementable by the image processing server70 include, for example, a transmission/reception unit 71, adetermination unit 72, a data processing unit 73, a storing/reading unit74, and a storage unit 7000.

The transmission/reception unit 71 transmits and receives various dataor information to and from another device (e.g., management server 90,display terminal 50, robot 10) via the communication network 9. Forexample, the transmission/reception unit 71 receives (acquires) thefull-view spherical image data 200 or the detailed image data 250 fromthe robot 10 (control device 30) via the communication network 9.Further, for example, the transmission/reception unit 71 receives(acquires) the state information 150 indicating the movement state ofthe robot 10 from the robot 10 (the control device 30) via thecommunication network 9. Further, for example, based on the receivedstate information 150, the transmission/reception unit 71 transmits(outputs) the full-view spherical image data 200 or the detailed imagedata 250, received from the robot 10 (the control device 30), to thedisplay terminal 50. The transmission/reception unit 71 is mainlyimplemented by the CPU 901 and the network I/F 909 of FIG. 15. Thetransmission/reception unit 71 is an example of a first acquisition unitin this description. Further, the transmission/reception unit 71 is anexample of a second acquisition unit in this description. Further, thetransmission/reception unit 71 is an example of a second transmissionunit in this description. Further, the transmission/reception unit 71 isan example of a second reception unit in this description.

Based on the state information 150 received (acquired) by thetransmission/reception unit 71, the determination unit 72 determines aspecific process to be performed on the full-view spherical image data200, or a specific process to be requested to the robot 10. Thedetermination unit 72 is mainly implemented by processing performed bythe CPU 901 of FIG. 15.

The data processing unit 73 has a function of performing a process ofgenerating a request command, which is an execution request to execute aspecific process on the full-view spherical image data 200 or a requestcommand to perform a specific process at the robot 10 based on the stateinformation 150 received (acquired) by the transmission/reception unit71. For example, the data processing unit 73 generates the requestcommand, which is an imaging request to the general imaging apparatus 23included in the robot 10. The data processing unit 73 is mainlyimplemented by processing performed by the CPU 901 of FIG. 15.

The storing/reading unit 74 stores various data in the storage unit 7000or reads various kinds of data from the storage unit 7000. Thestoring/reading unit 74 is mainly implemented by processing performed bythe CPU 901 of FIG. 15. The storage unit 7000 is implemented mainly bythe ROM 902, the HD 904, and the recording medium 906 of FIG. 15.

Further, the storage unit 7000 stores the full-view spherical image data200 and the detailed image data 250 received by thetransmission/reception unit 71. Further, the storage unit 7000 stores,for example, a state management table 7001, a condition table 7002, anda user command table 7003. Since the state management table 7001, thecondition table 7002 and the user command table 7003 are the same as thestate management table 5001 (FIG. 18), the condition table 5002 (FIG.19) and the user command table 5003 (FIG. 20), the description thereofwill be omitted. Further, the full-view spherical image data 200 and thedetailed image data 250 stored in the storage unit 7000 can be deletedwhen a specific time has elapsed after receiving the data by thetransmission/reception unit 71, or the full-view spherical image data200 and the detailed image data 250 can be deleted when a specific timeelapses after the transmission/reception unit 71 has transmitted(output) the full-view spherical image data 200 and the detailed imagedata 250 to the display terminal 50.

Processing and Operation in Third Embodiment:

Hereinafter, a description is given of the operation and processing ofthe remote-control system 1 c according to the third embodiment withreference to FIGS. 42 and 43. In FIG. 42 and FIG. 43, the processingperformed by the control device 30 included in the robot 10 will bedescribed as the processing performed by the robot 10. FIG. 42 is anexample of a sequence diagram illustrating the processing when the robot10 moves in an environment of the remote-control system 1 c according tothe third embodiment.

In steps S201-1, S201-2, and S201-3, the transmission/reception unit 31of the robot 10 transmits the full-view spherical image data 200, whichis the full-view spherical image captured by the special imagingapparatus 21, to the display terminal 50 using a communication sessionestablished with the management server 90. Then, thetransmission/reception unit 51 of the display terminal 50 receives thefull-view spherical image data 200.

In step S202, the display control unit 53 of the display terminal 50displays, on the display 512, the full-view spherical image data 200,received by the transmission/reception unit 51, as the display screen600 a (FIG. 27). With this configuration, the operator who operates therobot 10 using the display terminal 50 can confirm the status of thesite where the robot 10 is located by viewing the display screen 600 adisplaying the full-view spherical image data 200.

In step S203, the robot 10 moves within the site based on a requestcommand transmitted from the display terminal 50. In this case, themovement control unit 38 of the robot 10 controls the movement mechanism17 based on the request command transmitted from the display terminal50.

In step S204, the state information generation unit 35 of the robot 10generates the state information 150 (FIG. 28) based on the movementstate of the robot 10 acquired from the movement control unit 38.

In steps S205-1 and S205-2, the transmission/reception unit 31 of therobot 10 transmits the state information 150 generated by the stateinformation generation unit 35 to the image processing server 70 usingthe communication session established with the management server 90.Then, the transmission/reception unit 71 of the image processing server70 receives (acquires) the state information 150 from the robot 10 viathe management server 90 (an example of acquisition step).

In step S206, the data processing unit 73 of the image processing server70 performs the image processing on the full-view spherical image data200 received in step S201-2 based on the state information 150 received(acquired) by the transmission/reception unit 71 in step S205-2.

Hereinafter, a description is given of the image processing in the imageprocessing server 70, FIG. 43 is an example of a flowchart illustratingthe image processing on the full-view spherical image data based on themovement state of the robot 10 in the image processing server 70according to the third embodiment.

In step S206-1, if the transmission/reception unit 71 of the imageprocessing server receives (acquires) the state information 150transmitted from the robot 10 (5206-1: YES), the sequence proceeds tostep S206-2. On the other hand, if the transmission/reception unit 71does not receive the state information 150 (5206-1: NO), thetransmission/reception unit 71 repeats step S206-1.

In step S206-2, the storing/reading unit 74 of the image processingserver 70 reads out the condition table 7002 stored in the storage unit7000.

In step S206-3, the determination unit 72 of the image processing server70 searches the condition table 7002 read out by the storing/readingunit 74, and if there is at least one variable that satisfies theconditions included in the condition table 7002 among the variablesincluded in the state information 150 received by thetransmission/reception unit 71 (S206-3: YES), the determination unit 72proceeds the sequence to step S206-4. For example, when the numericalvalue corresponding to the variable name of “SPEED” included in thestate information 150 is “3.0 km/h,” the determination unit 72determines that “SPEED≤5.0 km/h,” “SPEED>1.0 km/h” and “SPEED>2.0 km/h”satisfy the condition among the items included in the condition table7002.

On the other hand, in step S206-3, if the determination unit 72determines that none of the variables included in the state information150 received by the transmission/reception unit 71 satisfies thecondition included in the condition table 7002 (S206-3: NO), thedetermination unit 72 terminates or end the sequence.

In step S206-4, the data processing unit 73 of the image processingserver 70 specifies a specific process corresponding to the concernedvariable item determined in step S206-3 and then performs the specifiedprocess on the full-view spherical image data 200. For example, the dataprocessing unit 73 specifies a specific process corresponding to each of“SPEED≤5.0 km/h,” “SPEED>1.0 km/h” and “SPEED>2.0 km/h” satisfying thecondition included in the condition table 7002. Then, the dataprocessing unit 73 of the image processing server 70 performs thespecific process (e.g., execution of process). In this case, the dataprocessing unit 73 returns the frame rate of the full-view sphericalimage data 200 to the initial state and performs the process of reducingthe resolution to 50%. Further, the data processing unit 73 alsogenerates an image that is zoomed out (ZOOM=−10) compared to the initialstate of the full-view spherical image data 200. Further, the dataprocessing unit 73 can be configured to perform all of the processescorresponding to the items satisfying the conditions, or execute one ormore processing corresponding to the items selected (extracted) from theitems satisfying the condition.

The description returns to FIG. 42. In step S207, thetransmission/reception unit 71 of the image processing server 70transmits the data processed by the data processing unit 73 (i.e.,processed full-view spherical image data 200) and the state information150 transmitted from the robot 10 to the display terminal 50. Then, thetransmission/reception unit 51 of the display terminal 50 receives theprocessed data (i.e., processed full-view spherical image data 200) andthe state information 150.

In step S208, the storing/reading unit 57 of the display terminal 50stores the state information 150, received by the transmission/receptionunit 51, in the state management table 5001 (FIG. 18) stored in thestorage unit 5000. Specifically, the storing/reading unit 57 of thedisplay terminal 50 stores one or more numerical values corresponding tothe one or more variable names included in the received stateinformation 150 for the state management table 5001.

In step S209, the display control unit 53 of the display terminal 50displays the movement state and the processed data (i.e., processedfull-view spherical image data 200) of the robot 10 on the displayscreen 600 (e.g., display screen 600 b illustrated in FIG. 29A).

Hereinafter, a description is given of a process of switching the imagedisplay between the full-view spherical image and the detailed image atthe display terminal 50 using the image processing server 70 withreference to FIG. 44. FIG. 44 is an example of a sequence diagramillustrating a process of switching the image displayed on the displayterminal 50 in an environment of in the remote-control system 1 caccording to the third embodiment. FIG. 44 illustrates an example casethat the display screen 600 a (displaying full-view spherical image data200) illustrated in FIG. 27 is displayed on the display terminal 50 inthe same manner in FIG. 39.

In step S101 c-1 and S101 c-2, the transmission/reception unit 31 of therobot 10 transmits the state information 150 (FIG. 28) generated by thestate information generation unit 35 to the image processing server 70using a communication session established with the management server 90.Then, the transmission/reception unit 71 of the image processing server70 receives (acquires) the state information 150 from the robot 10 viathe management server 90 (an example of acquisition step).

In step S102 c, if the robot 10 is stopped (S102 c: YES), the imageprocessing server 70 proceeds the sequence to step S103 c. Specifically,the determination unit 72 of the image processing server 70 determinesthat the robot 10 is stopped if the value of the variable of “SPEED”included in the state information 150 received by transmission/receptionunit 71 is “0 km/h.” It should be noted that the numerical value of thevariable of “SPEED” used for determining that the robot 10 is stopped isnot limited to “0 km/h.” For example, the determination unit 72 of theimage processing server 70 can be configured to determine that the robot10 is stopped if the numerical value of the variable of “SPEED” is lessthan or equal to a specific threshold value.

On the other hand, in step S102 c, if the determination unit 72determines (detects) that the robot 10 is not stopped but is moving(S102 c: NO), the determination unit 72 terminates or ends the sequence.

In step S103 c, the data processing unit 73 of the image processingserver 70 generates a request command including the “ARM” command andthe “SHOOT” command. As illustrated in the command table 3001 (FIG.17A), the processing of the robot 10 corresponding to the “SHOOT”command is “imaging by the general imaging apparatus 23.” That is, thedata processing unit 73 generates a control signal indicating an imagingrequest to the general imaging apparatus 23. Further, the variable (h,v)of the “ARM” command included in the request command is the directionthat the robot 10 is facing currently.

Specifically, the data processing unit 73 specifies the variable (h,v)using the information on the traveling direction included in the stateinformation 150 received (acquired) by the transmission/reception unit71 when the robot 10 was moving most recently. Further, the variable(h,v) of the “ARM” command included in the request command maycorrespond to the center position on the screen in the first imagecapture direction of the full-view spherical image data 200 displayed onthe display screen 600 a. Further, the variable (h,v) of the “ARM”command may correspond to a position specified by the mouse pointer 620on the screen in the first image capture direction of the full-viewspherical image data 200 displayed on the display screen 600 a. In thiscase, it is assumed that the image processing server 70 acquiresinformation on the display state of the display screen 600 a from thedisplay terminal 50.

In steps S104 c-1 and S104 c-2, the transmission/reception unit 71 ofthe image processing server 70 transmits the request command (“ARM” and“SHOOT”) generated by the data processing unit 73 to the robot 10. Inthis case, since the request command generated by the data processingunit 73 includes the “SHOOT” command, the transmission/reception unit 71transmits (outputs) a control signal indicating an imaging request tothe general imaging apparatus 23. Then, the transmission/reception unit31 of the robot 10 receives the request command (“ARM” and “SHOOT”) fromthe image processing server 70 via the management server 90.

In step S105 c, the arm operation control unit 39 of the robot 10transforms the manipulation arm 11 based on the request command receivedby the transmission/reception unit 31. The arm operation control unit 39changes the direction or orientation of the manipulation arm 11 bytransforming the manipulation arm 11 based on the “ARM” command includedin the request command received by the transmission/reception unit 31.Since the process of changing the direction or orientation of themanipulation ann 11 by the ann operation control unit 39 is the same asthe processing in step S93 of FIG. 34, the description thereof will beomitted.

In step S106 c, the imaging instruction unit 36 of the robot 10 outputsan instruction information indicating an imaging instruction to thegeneral imaging apparatus 23 in response to a completion of changing thedirection or orientation of the manipulation arm 11 by the arm operationcontrol unit 39. Then, the general imaging apparatus 23 performs theimaging process in response to the imaging instruction output from theimaging instruction unit 36. In this case, the general imaging apparatus23 acquires the detailed image data 250, which is a detailed imagecaptured in the direction or orientation of the manipulation arm 11changed by the arm operation control unit 39.

In step S107 c, the image acquisition unit 37 of the robot 10 acquiresthe detailed image data 250 captured by the general imaging apparatus23, from the general imaging apparatus 23.

In step S108 c, the storing/reading unit 41 of the robot 10 stores thedetailed image data 250 acquired by the image acquisition unit 37 in thestorage unit 3000.

In steps S109 c-1, S109 c-2 and S109 c-3, the transmission/receptionunit 31 of the robot 10 transmits the detailed image data 250 acquiredby the image acquisition unit 37 to the display terminal 50 via theserver system 7. Then, the transmission/reception unit 51 of the displayterminal 50 receives the detailed image data 250 transmitted from therobot 10. The transmission/reception unit 71 of the image processingserver 70 receives the detailed image data 250 transmitted from therobot 10 in step S109 c-2. Then, the transmission/reception unit 71 ofthe image processing server 70 outputs the received detailed image data250 to the display terminal 50 in step S109 c-3. That is, the imageprocessing server 70 outputs the detailed image data 250 based on thestate information 150 acquired in step S101 c-2. It should be noted thatthe processing order of step S108 c and steps S109 c-1, S109 c-2 andS109 c-3 can be changed or can be performed in parallel with each other.

In step S110 c, the storing/reading unit 57 of the display terminal 50stores the detailed image data 250 received by thetransmission/reception unit 51 in the storage unit 5000.

In step S111 c, the display control unit 53 of the display terminal 50displays the detailed image data 250 received by thetransmission/reception unit 51 on the display 512. With thisconfiguration, the display control unit 53 switches the image displayedon the display 512 from the full-view spherical image data 200 (e.g.,display screen 600 a in FIG. 27) to the detailed image data 250 (e.g.,display screen 650 in FIG. 35). It should be noted that the order ofprocessing in step S110 c and step S111 c can be changed or can beperformed in parallel with each other.

Further, the image processing server 70 can be configured to generatethe request command if the determination unit 72 has determined that therobot 10 is stopped in step S102 c and then the determination unit 72determines or detects that the robot 10 is not stopped (i.e., the robot10 is moving again) after the determination unit 72 has determined thatthe robot 10 is stopped in step S102 c.

Further, the image processing server 70 can be configured to generatethe request command if the determination unit 72 has determined that therobot 10 is stopped in step S102 c and then a given time period elapsesafter the determination unit 72 has determined that the robot 10 isstopped in step S102 c.

With this configuration, even if the robot 10 is being stopped at thesame position, the remote-control system 1 c can reduce the probabilitythat the detailed image data 250 is acquired by the general imagingapparatus 23 and then displayed on the display terminal 50, with whichwasteful processing such as acquiring the detailed image data 250 whilethe robot 10 is being stopped at the same position can be reduced.Further, with this configuration, the remote-control system 1 c canreduce the processing load of the robot 10 and the display terminal 50and prevent the occurrence of communication delay caused by thecongestion of the communication network 9.

As to the above described remote-control system 1 c according to thethird embodiment, the image processing server 70 (an example ofinformation processing apparatus) determines the image (i.e., thefull-view spherical image data 200 or the detailed image data 250) to betransmitted (output) to the display terminal 50 based on the movementstate of the robot 10 (an example of movable apparatus). Therefore, asto the remote-control system 1 c, by transmitting an appropriate imagebased on the movement state of the robot 10 from the image processingserver 70 to the display terminal 50, occurrence of time lag of thedisplay screen due to communication delay can be prevented, and bydisplaying an image suitable for the operator of the robot 10 on thedisplay terminal 50, the remote control of the robot 10 including thespecial imaging apparatus 21 and the general imaging apparatus 23 can beimplemented with higher accuracy.

Further, the image processing server 70 (an example of informationprocessing apparatus) according to the third embodiment can communicatewith, via the communication network 9, the display terminal 50 thatcontrols the robot 10 (an example of movable apparatus) including thespecial imaging apparatus 21 (an example of first imaging apparatus),which captures images of objects and acquires the full-view sphericalimage data 200 (an example of first image), and the general imagingapparatus 23 (an example of second imaging apparatus), which captures apart or portion of the objects captured by the special imaging apparatus21 (an example of second image).

Further, based on the movement state of the robot 10, the imageprocessing server 70 transmits the full-view spherical image data 200 orthe detailed image data 250, received from the special imaging apparatus21 or the general imaging apparatus 23 to the display terminal 50 toswitch the imaged displayed on the display terminal 50 between thefull-view spherical image data 200 and the detailed image data 250.Therefore, the image processing server 70 can reduce the processing loadof the robot 10 and the display terminal 50, and can implement advancedcontrol when the remote control is performed using the robot 10 equippedwith the special imaging apparatus 21 and the general imaging apparatus23.

In the remote-control system 1 c according to the third embodiment, theimage processing server 70 is an example of an output control apparatusaccording to the embodiment of the present invention. The imageprocessing server 70 (an example of information processing apparatus)used as the output control apparatus controls the robot 10 (an exampleof movable apparatus) including the special imaging apparatus 21 (anexample of first imaging apparatus), which captures images of objectsand acquires the full-view spherical image data 200 (an example of firstimage), and the general imaging apparatus 23 (an example of secondimaging apparatus), which captures a part or portion of the objectscaptured by the special imaging apparatus 21 (an example of secondimage). Then, the image processing server 70 used as the output controlapparatus acquires (receives) the state information 150 indicating themovement state of the robot 10, and outputs (transmits) the full-viewspherical image data 200 and the detailed image data 250 selectivelybased on the acquired state information 150. Therefore, the imageprocessing server 70 used as the output control apparatus can implementadvanced control when the remote control is performed on the robot 10equipped with the special imaging apparatus 21 and the general imagingapparatus 23.

Further, as to the remote-control system 1 c according to the thirdembodiment, the image processing server 70 is an example of imagingcontrol apparatus according to the present invention. The imageprocessing server 70 (an example of information processing apparatus)used as the imaging control apparatus controls the robot 10 (an exampleof movable apparatus) including the special imaging apparatus 21 (anexample of first imaging apparatus), which captures images of objectsand acquires the full-view spherical image data 200 (an example of firstimage), and the general imaging apparatus 23 (an example of secondimaging apparatus), which captures a part or portion of the objectscaptured by the special imaging apparatus 21 (an example of secondimage). Then, the image processing server 70 used as the imaging controlapparatus acquires (receives) the state information 150 indicating themovement state of the robot 10 and outputs (transmits) the instructioninformation (an example of control signal indicating imaging request) toinstruct the imaging request to the general imaging apparatus 23 basedon the acquired state information 150. Therefore, the image processingserver 70 used as the imaging control apparatus can implement advancedcontrol when the remote control is performed using the robot 10 equippedwith the special imaging apparatus 21 and the general imaging apparatus23.

As above described, the remote-control systems 1 a and 1 b according tothe first and second embodiments include the robot 10 (an example ofmovable apparatus) including the special imaging apparatus 21 (anexample of first imaging apparatus), which captures images of objectsand acquires the full-view spherical image data 200 (an example of firstimage), and the general imaging apparatus 23 (an example of secondimaging apparatus), which captures a part or portion of the objectscaptured by the special imaging apparatus 21 (an example of secondimage), and the display terminal 50 that can communicate with the robot10 via the communication network 9 and can perform the remote control ofthe robot 10. Then, the remote-control systems 1 a and 1 b acquire thestate information 150 indicating the movement state of the robot 10, andoutput the full-view spherical image data 200 and the detailed imagedata 250 selectively based on the acquired state information 150.Therefore, the remote-control systems 1 a and 1 b can implement theremote control of the robot 10 equipped with the special imagingapparatus 21 and the general imaging apparatus 23 with higher accuracy.

As to the above described embodiments of the present invention, thecontrol of the movable apparatus equipped with the imaging apparatus,such as the special imaging apparatus 21 and the general imagingapparatus 23, can be performed with higher accuracy or precision.

Further, the remote-control system 1 c according to the third embodimentincludes the image processing server 70 (an example of informationprocessing apparatus) that can communicate with the robot 10 and thedisplay terminal 50 via the communication network 9. Therefore, theremote-control system 1 c can reduce the processing load of the robot 10and the display terminal 50, and can implement advanced control when theremote control is performed using the robot 10 equipped with the specialimaging apparatus 21 and the general imaging apparatus 23.

The functions of each of the embodiments can be implemented by computerexecutable programs such as programs described in legacy programminglanguages and object oriented programming languages such as Assembler,C++, C++, Java (registered trademark) and the like, and stored on acomputer-readable recording medium such as ROM, electrically erasableprogrammable read-only memory (EEPROM), erasable programmable read-onlymemory (EPROM), flash memory, flexible disk, CD-ROM, compactdisc-rewritable (CD-RW), digital versatile disc (DVD)-ROM, DVD-RAM,DVD-RW, Blu-ray disc, secure digital (SD) card, magneto-optical disc(MO), or the like, and the computer executable program is distributablevia telecommunication lines.

Further, some or all of the functions of each embodiment can beimplemented on programmable devices (PD) such as field programmable gatearray (FPGA) or the like, or can be implemented as application specificintegrated circuits (ASIC), and the programs can be distributed usingrecording media as data described in HDL (Hardware DescriptionLanguage), VHDL (Very high speed integrated circuits hardwaredescription language), Verilog HDL and the like used for generating thecircuit configuration data (bit stream data), and downloaded to the PDto implement the functions of each of the embodiments on the PD.

The output control apparatus, the display terminal, the informationprocessing apparatus, the movable apparatus, the remote-control system,the output control method, the storage medium of program and the imagingcontrol apparatus according to the embodiments of the present inventionhave been described, but the present invention is not limited to theembodiments described above. Each of the embodiments described above ispresented as an example, and it is not intended to limit the scope ofthe present invention. Numerous additional modifications and variationsare possible in light of the above teachings. It is therefore to beunderstood that, within the scope of the appended claims, the disclosureof this specification can be practiced otherwise than as specificallydescribed herein. Any one of the above-described operations may beperformed in various other ways, for example, in an order different fromthe one described above. Each of the functions of the describedembodiments may be implemented by one or more processing circuits orcircuitry. Processing circuitry includes a programmed processor, as aprocessor includes circuitry. A processing circuit also includes devicessuch as an application specific integrated circuit (ASIC), digitalsignal processor (DSP), field programmable gate array (FPGA), andconventional circuit components arranged to perform the recitedfunctions.

What is claimed is:
 1. A control apparatus for controlling a movableapparatus including a first imaging apparatus for imaging a target toacquire a first image and a second imaging apparatus for imaging a partof the target to acquire a second image, the control apparatuscomprising: circuitry configured to acquire, from the movable apparatus,state information indicating a movement state of the movable apparatus;receive the first image and the second image from the movable apparatusbased on the acquired state information; and output the first image andthe second image selectively based on the acquired state information. 2.The control apparatus according to claim 1, wherein when the circuitrydetects that the movable apparatus s stopped, the circuitry outputs thesecond image.
 3. The control apparatus according to claim 1, wherein thestate information includes information on a traveling velocity of themovable apparatus, wherein when the circuitry detects that the travelingvelocity of the movable apparatus becomes a threshold value or less, thecircuitry outputs the second image.
 4. The control apparatus accordingto claim 1, wherein the circuitry instructs the first imaging apparatusof the movable apparatus to change image quality of the first imagebased on the state information acquired from the movable apparatus, andreceives the first image having the changed image quality from themovable apparatus, and the image quality includes at least one of aframe rate, a resolution and an output range of the first image.
 5. Thecontrol apparatus according to claim 4, wherein the state informationincludes information on a traveling velocity of the movable apparatus,wherein when the circuitry detects that the traveling velocity of themovable apparatus becomes a threshold value or less, the circuitryinstructs the first imaging apparatus of the movable apparatus to changethe image quality of the first image by setting a parameter of the imagequality, and receive the first image having the changed image qualityfrom the movable apparatus, wherein the parameter of the image qualityis set to increase the frame rate when the image quality includes theframe rate, the parameter of the image quality is set to increase theresolution when the image quality includes the resolution, and theparameter of the image quality is set to a smaller output range when theimage quality includes the output range.
 6. The control apparatusaccording to claim wherein the first image is a video image and thesecond image is a still image.
 7. The control apparatus according toclaim 1, wherein the control apparatus is a display terminalcommunicable with the movable apparatus via a communication network forremotely operating the movable apparatus, wherein the display terminalincludes circuitry is configured to receive the first image and thesecond image from the movable apparatus, and the circuitry of thedisplay terminal is configured to switch an image displayed on a displaybetween the first image and the second image based on the acquired stateinformation.
 8. The control apparatus according to claim 7, wherein whenthe circuitry of the display terminal detects that the movable apparatusis stopped, the circuitry of the display terminal displays the secondimage on the display.
 9. The control apparatus according to claim 7,wherein the state information includes information on a travelingvelocity of the movable apparatus, wherein when the circuitry detectsthat the traveling velocity of the movable apparatus becomes a thresholdvalue or less, the circuitry of the display terminal displays the secondimage on the display.
 10. The control apparatus according to claimwherein the circuitry of the display terminal generates an imagingrequest to be requested to the second imaging apparatus based on theacquired state information and transmits the imaging request to themovable apparatus, wherein the circuitry of the display terminalreceives, from the movable apparatus, the second image acquired by thesecond imaging apparatus based on the imaging request transmitted to themovable apparatus, and displays the received second image on thedisplay.
 11. The control apparatus according to claim 10, wherein theimaging request to be requested to the second imaging apparatus includesinformation indicating a second image capture direction to be used bythe second imaging apparatus, and the circuitry of the display terminaldetermines the second image capture direction to be used by the secondimaging apparatus based on a first image capture direction used for thefirst image being displayed on the display of the display terminal. 12.The control apparatus according to claim 7, wherein the circuitry of thedisplay terminal instructs the first imaging apparatus of the movableapparatus to change image quality of the first image based on the stateinformation acquired from the movable apparatus, and displays the firstimage having the changed image quality on the display of the displayterminal, and the image quality includes at least one of a frame rate, aresolution and an output range of the first image.
 13. The controlapparatus according to claim 12, wherein the state information includesinformation on a traveling velocity of the movable apparatus, whereinwhen the circuitry of the display terminal detects that the travelingvelocity of the movable apparatus becomes a threshold value or less, thecircuitry of the display terminal instructs the first imaging apparatusof the movable apparatus to change the image quality of the first imageby setting a parameter of the image quality, and receives and displaysthe first image having the changed image quality on the display of thedisplay terminal. wherein the parameter of the image quality is set toincrease the frame rate when the image quality includes the frame rate,the parameter of the image quality is set to increase the resolutionwhen the image quality includes the resolution, and the parameter of theimage quality is set to a smaller output range when the image qualityincludes the output range.
 14. The control apparatus according to claim1, wherein the control apparatus is an information processing apparatuscommunicable with a display terminal used for remotely operating themovable apparatus via a communication network, wherein the informationprocessing apparatus includes circuitry configured to acquire the firstimage and the second image from the movable apparatus and transmit theacquired first image and second image to the display terminal, whereinthe circuitry of the information processing apparatus transmits theacquired first image and second image selectively to the displayterminal based on the acquired state information to switch an imagedisplayed on the display terminal between the first image and the secondimage.
 15. A movable apparatus comprising: a movement mechanismconfigured to move the movable apparatus; a first imaging apparatus forimaging a target to acquire a first image; a second imaging apparatusfor imaging a part of the target to acquire a second image; andcircuitry configured to acquire state information indicating a movementstate of the movable apparatus; acquire the first image using the firstimaging apparatus and the second image using the second imagingapparatus; and output the acquired first image and second image to adisplay terminal, communicable with the circuitry, based on the acquiredstate information.
 16. The movable apparatus according to claim 15,wherein when the circuitry detects that the movable apparatus isstopped, the circuitry transmits the second image to the displayterminal to display the second image on the display terminal.
 17. Themovable apparatus according to claim 15, wherein the state informationincludes information on a traveling velocity of the movable apparatus,wherein when the circuitry detects that the traveling velocity of themovable apparatus becomes a threshold value or less, the circuitrytransmits the second image to the display terminal to display the secondimage on the display terminal.
 18. The movable apparatus according toclaim 15, wherein the state information includes information on atraveling velocity of the movable apparatus, wherein when the circuitrydetects that the traveling velocity of the movable apparatus becomes athreshold value or less, the circuitry receives the first image,acquired by the first imaging apparatus by increasing image quality ofthe first image, from the first imaging apparatus, and transmits thefirst image having the increased image quality to the display terminalto display the first image having the increased image quality on thedisplay terminal.
 19. A remote-control system comprising: a movableapparatus including a first imaging apparatus for imaging a target toacquire a first image and a second imaging apparatus for imaging a partof the target to acquire a second image; a display terminal communicablewith the movable apparatus via a communication network for remotelyoperating the movable apparatus; and circuitry configured to acquire,from the movable apparatus, state information indicating a movementstate of the movable apparatus; receive the first image and the secondimage from the movable apparatus based on the acquired stateinformation; and output, to the display terminal, the first image andthe second image selectively based on the acquired state information.